Neural network model clipping method, device, electronic device and storage medium

Abstract

The present application provides a neural network model pruning method, device, electronic device, and storage medium. The method includes: dividing the neural network model to be pruned into multiple structural groups based on the structural characteristics of the neural network model to be pruned; determining a target pruning rate for pruning each structural group in the multiple structural groups based on the change in the loss function before and after pruning the neural network model to be pruned; wherein the change in the loss function is determined based on a first factor and a second factor, the first factor being used to characterize the importance of each weight in each structural group, and the second factor being used to characterize the interaction between different structural groups; and pruning each structural group based on the target pruning rate to obtain a pruned neural network model. This method can accelerate the compression of neural network models with minimal performance loss.

Description

Technical Field

[0001] The present application relates to the field of chips and neural network compression, and in particular to a neural network model clipping method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, the use of neural network models to perform intelligent tasks, such as intelligent detection (e.g., vehicle and license plate detection), is becoming increasingly popular. Since terminal devices typically have limited computing resources, overly complex neural network models can place excessive demands on computing resources, resulting in slower execution of intelligent tasks and poor real-time performance. Therefore, to improve the real-time performance of intelligent tasks performed by terminal devices, the neural network models used on these devices can be tailored appropriately.

[0003] How to efficiently and reasonably trim neural network models has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In view of this, the present application provides a neural network model pruning method, device, electronic device and storage medium to accelerate the compression of the neural network model while reducing performance loss.

[0005] Specifically, this application is implemented through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a method for pruning a neural network model is provided, comprising:

[0007] Dividing the neural network model to be pruned into a plurality of structural groups according to structural characteristics of the neural network model to be pruned;

[0008] Determining a target cropping rate for cropping each of the multiple structural groups based on a change in the loss function before and after cropping the neural network model to be cropped; wherein the change in the loss function is determined based on a first factor and a second factor, the first factor being used to characterize the importance of each weight in each structural group, and the second factor being used to characterize the interaction between different structural groups;

[0009] According to the target cropping rate, each structure group is cropped separately to obtain a cropped neural network model.

[0010] According to a second aspect of an embodiment of the present application, a neural network model pruning device is provided, comprising:

[0011] A division unit, configured to divide the neural network model to be pruned into a plurality of structural groups according to structural characteristics of the neural network model to be pruned;

[0012] a determining unit, configured to determine a target cropping rate for cropping each of the plurality of structural groups based on a change in a loss function before and after cropping the neural network model to be cropped; wherein the change in the loss function is determined based on a first factor and a second factor, the first factor being used to characterize the importance of each weight in each structural group, and the second factor being used to characterize the interaction between different structural groups;

[0013] The cropping unit is used to crop each structure group according to the target cropping rate to obtain a cropped neural network model.

[0014] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the method provided in the first aspect.

[0015] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0016] The technical solution provided by this application can at least bring the following beneficial effects:

[0017] The neural network model to be pruned is divided into multiple structural groups according to its structural characteristics, and the target cropping rate for cropping each structural group is determined according to the change of the loss function before and after the pruning of the neural network model to be pruned. Then, each structural group is pruned according to the target cropping rate to obtain the pruned neural network model. On the one hand, the neural network model to be pruned is divided into structural groups and the cropping rate is searched based on the structural groups; on the other hand, when determining the change of the loss function before and after the pruning of the neural network model to be pruned, the importance of each weight in each structural group and the interaction between different structural groups are considered, thereby improving the rationality of the determined cropping rate of each structural group, and accelerating the compression of the neural network model while reducing performance loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of a neural network model pruning method shown in an exemplary embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a visual Transform model for image classification shown in an exemplary embodiment of the present application;

[0020] Figure 3This is a flowchart of a neural network model pruning process shown in an exemplary embodiment of the present application;

[0021] Figure 4 1 is a schematic diagram of dividing a Hessian matrix into blocks according to different structural objects, as shown in an exemplary embodiment of the present application;

[0022] Figure 5 1 is a schematic structural diagram of a neural network model cutting device shown in an exemplary embodiment of the present application;

[0023] Figure 6 It is a schematic diagram of the hardware structure of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0025] The terms used in this application are for the purpose of describing particular embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0026] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0027] See Figure 1 , is a flow chart of a neural network model pruning method provided in an embodiment of the present application, such as Figure 1 As shown, the neural network model pruning method may include the following steps:

[0028] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0029] Step S100: Divide the neural network model to be pruned into a plurality of structural groups according to the structural characteristics of the neural network model to be pruned.

[0030] In the embodiments of the present application, considering that different structures in a neural network model generally have different functions, if the entire neural network model is pruned, at the same pruned rate, the performance loss of the pruned neural network model may be relatively large, and the performance of the pruned neural network model cannot be guaranteed.

[0031] Therefore, in order to optimize the performance of the pruned neural network model, the neural network model to be pruned can be divided into multiple units, and each unit is used as a pruned object to pruned the neural network model.

[0032] Accordingly, for any neural network model to be pruned, the neural network model to be pruned can be divided into multiple structural groups according to the structural characteristics of the neural network model to be pruned.

[0033] In one example, the neural network model to be pruned includes a Transform model, and the above-mentioned multiple structure groups may include: a head in a self-attention structure, a fully connected layer in a feedforward neural network, and a fully connected layer constrained by residual connections.

[0034] In another example, the neural network model to be trimmed includes a convolutional neural network model. When the convolutional neural network model is structurally grouped, different network layers can be divided into different structural groups, or different stages of the convolutional neural network can be divided into different structural groups.

[0035] Step S110: Determine a target cropping rate for cropping each of the multiple structural groups based on a change in the loss function before and after cropping the neural network model to be cropped; wherein the change in the loss function is determined based on a first factor and a second factor, the first factor is used to characterize the importance of each weight in each structural group, and the second factor is used to characterize the interaction between different structural groups.

[0036] In an embodiment of the present application, in order to ensure the performance of the pruned neural network model, in the process of determining the pruned rate of the neural network model, the performance loss of the pruned neural network model relative to the pre-pruned neural network model can be determined based on the change in the loss function of the neural network model before and after pruned.

[0037] For example, the greater the change in the loss function, the greater the performance loss of the pruned neural network model relative to the unpruned neural network model.

[0038] In addition, considering that when pruning each structural grouping, the pruning of any structural grouping will not only affect the processing performance of the structural grouping itself, but also affect the processing performance of other structural groupings, therefore, when pruning structural groupings, it is necessary to consider not only the importance of the individual structure itself, but also the interaction between different structural groupings to avoid a serious degradation in the performance of the neural network model after pruning.

[0039] Accordingly, when determining the change in the loss function of the neural network model before and after pruning, it is necessary to consider the influence and interaction of each structure in the above-mentioned multiple structural groupings, model the interaction between different structural groupings, and obtain a first factor for characterizing the importance of each weight in a single structural grouping, and a second factor for characterizing the interaction between different structural groupings. For example, the importance of a single weight is used to characterize the effect of each weight itself on the network (i.e., the effect of pruning a single weight in a structural grouping on the loss of model performance), and the interaction between different structural groupings is used to characterize the effect of the correlation between structural groupings on the network (i.e., the effect of pruning multiple structural groupings on the loss of model performance). The change in the loss function of the neural network model before and after pruning is determined by combining the importance of a single weight and the interaction between different structural groupings.

[0040] In the process of pruning the neural network model to be pruned, the pruning rate (which may be referred to as the target pruning rate) for each of the multiple structural groupings can be determined based on the change in the loss function of the neural network model to be pruned before and after pruning.

[0041] Illustratively, the determined target cropping ratio includes a plurality of cropping ratios for respectively cropping the plurality of structure groups.

[0042] For example, assuming that the neural network model to be pruned is divided into three structural groups, the target pruned rate may include three pruned rates, which are respectively used to pruned the three structural groups.

[0043] Exemplarily, for any structural grouping, the pruning rate of the structural grouping may be the ratio of the number of weights that need to be pruned to the total number of weights of the structural grouping.

[0044] Exemplarily, after pruning the above-mentioned multiple structure groups according to the target pruning rate, the computational amount of the pruned neural network model can be consistent with the given computational amount target, for example, the same as the given computational amount target, or the deviation from the given computational amount target is within a preset range.

[0045] Step S120: Crop each structure group according to the target cropping rate to obtain a cropped neural network model.

[0046] In an embodiment of the present application, when the target cropping rate for cropping each structural grouping is determined in the above manner, each structural grouping can be cropped separately according to the determined target cropping rate for cropping each structural grouping to obtain a cropped neural network model (which can be called a sub-model).

[0047] For example, for any structural grouping, the number of weights that need to be trimmed can be determined based on the target trimming rate of the structural grouping, and trimming can be started from the least important weight according to the importance of each weight in the structural grouping until the number of trimmed weights meets the requirement.

[0048] It can be seen that in Figure 1 In the method flow shown, the neural network model to be pruned is divided into multiple structural groups according to the structural characteristics of the neural network model to be pruned, and the target cropping rate for cropping each structural group is determined according to the change of the loss function before and after the pruned neural network model. Then, according to the target cropping rate, each structural group is pruned respectively to obtain the pruned neural network model. On the one hand, the neural network model to be pruned is divided into structural groups and the cropping rate is searched with the structural groups as the object; on the other hand, when determining the change of the loss function before and after the pruned neural network model, the importance of each weight in each structural group and the interaction between different structural groups are considered, thereby improving the rationality of the determined cropping rate of each structural group, and accelerating the compression of the neural network model with less performance loss.

[0049] In some embodiments, determining the target cropping rate for cropping each of the multiple structure groups based on the change in the loss function before and after cropping the neural network model to be cropped may include:

[0050] Initializing a cropping rate population; wherein the cropping rate population includes a plurality of seeds, each seed including a cropping rate for cropping each of the plurality of structure groups;

[0051] According to the changes in the loss function corresponding to various seeds, the cropping rate population is iteratively updated until the number of iterations reaches the specified number of iterations, and the target cropping rate for cropping each of the multiple structure groups is determined based on the iteratively updated cropping rate population.

[0052] For example, in order to improve the rationality of the determined target cropping rate and further optimize the performance of the pruned neural network model, a population-based iterative optimization method can be used to search for the target cropping rate for pruning each structural group.

[0053] Exemplarily, when the neural network to be pruned is divided into multiple structural groups in the above manner, the pruned rate population can be initialized.

[0054] Illustratively, the cropping rate population may include multiple seeds, and one seed may include a cropping rate for cropping each structure group.

[0055] For any seed in the cropping rate population, each structure group can be cropped according to the cropping rate corresponding to the seed, and the loss function change of the neural network model before and after cropping (which can be called the loss function change corresponding to the seed) can be determined.

[0056] When the loss function changes corresponding to various seeds in the cropping rate population are determined, the cropping rate population can be iteratively updated according to the loss function changes corresponding to various seeds until the number of iterations reaches the specified number of iterations. The target cropping rate for cropping each structural group is determined based on the iteratively updated cropping rate population.

[0057] It should be noted that for any seed in the cropping rate population, after cropping the above-mentioned multiple structure groups according to the cropping rate corresponding to the seed, the computational amount of the pruned neural network model can be consistent with the given computational amount target, for example, the same as the given computational amount target, or the deviation from the given computational amount target is within a preset range.

[0058] In one example, the iterative updating of the crop rate population according to the change in the loss function corresponding to each seed may include:

[0059] For any iteration, the various seeds in the current crop rate population are sorted according to the change in the loss function corresponding to the various seeds in the current crop rate population;

[0060] Based on the M seeds with the smallest change in loss function in the current crop rate population, a new crop rate population is generated using an evolutionary algorithm; where M ≥ 1.

[0061] Illustratively, in the process of iteratively updating the cropping rate population, for any iteration, the various seeds in the current cropping rate population may be sorted according to the change in the loss function corresponding to the various seeds in the current cropping rate population.

[0062] For example, various seeds in the current cropping rate population are sorted in the order of corresponding loss function changes from low to high; or various seeds in the current cropping rate population are sorted in the order of corresponding loss function changes from high to low.

[0063] For example, a new cropping rate population may be generated using a crossover mutation method based on the M seeds with the smallest corresponding loss function changes in the current cropping rate population.

[0064] Taking the example of sorting the seeds in the current cropping rate population according to the order of the corresponding loss function changes from low to high, a new cropping rate population can be generated based on the first M seeds in the sorted cropping rate population using the crossover mutation method.

[0065] In some embodiments, the first factor is a single-weight perturbation factor, and the second factor is a weight-associated perturbation factor;

[0066] The change in the loss function before and after pruning the neural network model can be determined as follows:

[0067] Determining a change in the loss function before and after pruning the neural network model to be pruned based on the single weight perturbation factors of the plurality of structural groups and the weight association perturbation factors between the plurality of structural groups;

[0068] Among them, the single weight perturbation factor is used to characterize the performance loss of the neural network model before and after a single weight is pruned, and the weight association perturbation factor is used to characterize the performance loss of the neural network model before and after the associated weights are pruned.

[0069] For example, considering that there may be correlations between different weights in a neural network model, for example, for two weights in a neural network model, when either of the two weights is pruned individually, the performance loss of the neural network model before and after pruning may be relatively small. However, when both weights are pruned, the performance loss of the neural network model before and after pruning may be relatively large.

[0070] Accordingly, when determining the change in the loss function before and after pruning the neural network model, for the pruned weights, it is necessary to consider not only the impact of pruning each individual pruned weight on the performance of the neural network model, but also the correlation between the weights, that is, the impact of pruning multiple associated weights on the neural network model.

[0071] For example, in the process of searching for the optimal cropping rate for each structural grouping, the change in loss function before and after cropping the neural network model to be cropped can be determined based on the single weight perturbation factors of multiple structural groupings and the weight-related perturbation factors between multiple structural groupings. Then, based on the change in loss function before and after cropping the neural network model to be cropped, the optimal cropping rate for cropping each structural grouping can be determined.

[0072] Exemplarily, the single weight interference factor of the weight can be determined based on the weight gradient.

[0073] It should be noted that when performing weight pruning, there may be multiple weights that need to be pruned together.

[0074] For example, when performing weight pruning on the head in the self-attention structure of the Transform model, pruning needs to be performed on a head-by-head basis. However, a head usually includes multiple weights, that is, the multiple weights need to be pruned together.

[0075] Accordingly, for multiple weights that need to be pruned together, the single-weight interference factors corresponding to the multiple weights can be determined based on the impact of the pruning of the multiple weights on the performance of the neural network model.

[0076] Exemplarily, in the process of determining the weight interference factor between the structural groups, the Hessian matrix may be calculated and determined first, and the weight interference factor may be determined based on the Hessian matrix.

[0077] In one example, the weight association perturbation factors between the above-mentioned multiple structural groups are determined by:

[0078] For the first structural grouping and the second structural grouping among the above-mentioned multiple structural groupings, the weight association disturbance factor between the first structural grouping and the second structural grouping is determined based on the number of weights in the first structural grouping, the number of weights in the second structural grouping, the clipping rate of the first structural grouping, the clipping rate of the second structural grouping, the first matrix block and the second matrix block; wherein the first matrix block is the matrix block corresponding to the first structural grouping in the Hessian matrix, and the second matrix block is the matrix block corresponding to the second structural grouping in the Hessian matrix.

[0079] For example, considering that each time the weight-related perturbation factor between structural groups is calculated, the Hessian matrix needs to be calculated, which will occupy a lot of computing resources and affect the computing efficiency, the entire Hessian matrix can be divided into blocks according to different structural groups. Each structural group corresponds to a block in the Hessian matrix, and different structural objects interact to form cross terms.

[0080] Exemplarily, for any two structural groups among the above-mentioned multiple structural groups (respectively referred to as the first structural group and the second structural group), the weight association disturbance factor between the first structural group and the second structural group can be determined based on the number of weights in the first structural group, the number of weights in the second structural group, the clipping rate of the first structural group, the clipping rate of the second structural group, the first matrix block and the second matrix block. The specific implementation can be explained in conjunction with specific examples below.

[0081] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.

[0082] In this embodiment, the neural network model to be pruned is taken as an example as Transform model.

[0083] like Figure 2 As shown in, it is a schematic diagram of a visual Transform model for image classification, such as Figure 2 As shown in the figure, the input image is divided into multiple different image blocks through the patchify layer (patcchify stem), and then these image blocks are sent to the Transformer layer (i.e. Figure 2 The backbone model composed of the Transformer block in the image extracts image feature information, and finally inputs these feature information into the fully connected layer (also called the linear layer) for classification. In this embodiment, each Transformer layer contains a variety of heterogeneous structures, such as Figure 2 As shown in the figure, it includes two major substructures: a multi-head attention layer (MSA) and a feedforward neural network layer (FFN). The multi-head attention layer includes a normalization layer, a multi-head attention operation, a fully connected layer, and an add layer, while the feedforward neural network layer includes a normalization layer, a linear layer, and a summation layer.

[0084] For example, the entire network can be divided into three structural groups according to the structure: the first structural group includes the head of the multi-head attention layer (i.e., head), such as Figure 2 As shown in Figure 1, each head contains three linear layers and dot-product attention. Each head can extract different feature information, but if the number of heads is too large, the same information will be extracted.

[0085] Taking the dog image to be classified as an example, head 1 and head 2 both focus on the dog's hair texture information, so one head can be cropped off.

[0086] The second structure is grouped into fully connected layers in the feed-forward neural network structure, such as Figure 2 As shown, it can fuse information from different channels.

[0087] For example, consider the dog being classified. The multi-head attention layer has already learned some feature information, which can be represented as a feature tensor with multiple channels. Each channel contains different information, such as the characteristics of the dog's tail or eyes. Each neuron in the fully connected layer can fuse these channel features. If there are too many neurons, different neurons may fuse the same features. In this case, the redundant neurons can be pruned.

[0088] The third structure is grouped into two fully connected layers in substructures, such as Figure 2As shown in Figure 2, since these two structures need to maintain the same structure through residual connection constraints, they can be divided into the same structure group.

[0089] It can be seen that through the above-mentioned structural grouping method, the redundant structures of different structural groups can be automatically identified and pruned and compressed, which effectively reduces the number of parameters and calculations with a small performance loss.

[0090] For example, the computational complexity of the cropped Transform model can be specified by the user, and then the structure can be grouped and cropped according to the above method. According to the computational complexity specified by the user, a compressed model of corresponding size can be obtained. The entire technical solution is implemented quickly and efficiently, and the compressed model has high accuracy and good performance.

[0091] like Figure 3 As shown, the implementation process of the clipping solution of the Transform model provided in this embodiment may include:

[0092] 1. Structural grouping

[0093] For example, Figure 2 Taking the visual Tranformer model for image classification shown in FIG, it can be divided into different structural groups according to structure (a structural group can be called a structural object), and each structural object has its own cropping rate.

[0094] 2. Clipping rate evolution search

[0095] Exemplarily, an evolutionary search algorithm may be used to determine the most appropriate cropping rate (ie, the target cropping rate) for each structural object.

[0096] For example, when configuring the initial parameters of the evolutionary search, the cropping rate population may be randomly initialized, and each seed in the population represents the cropping rate of each object.

[0097] In each iteration, each seed in the population is pruned according to its pruning ratio. The resulting sub-model then calculates the change in loss function for that seed (also known as the approximate loss function change). All seeds are sorted based on this change, and the M seeds with the smallest approximate loss function change are selected as the next generation. A new population is generated using crossover mutation. This process continues until the set number of iterations is reached, at which point the optimal sub-model is found.

[0098] For example, for any structural object, when pruning the structural object, all weights in the structural object can be sorted according to the importance of each weight itself, and then pruned starting from the weight with the least importance according to a given pruning ratio to obtain a simpler structural object.

[0099] For example, the change in the approximate loss function before and after pruning the Transform model can be determined as follows:

[0100] Assuming the model loss function is L and the weight is W, after clipping it becomes W+ΔW, and the corresponding loss function changes to:

[0101] ΔL=L(W+ΔW)-L(W) (1)

[0102] The change in the loss function can reflect the impact on model performance. By expanding the loss function using Taylor, we can obtain a calculation formula that can be analytically quantified:

[0103]

[0104] The first term on the right side of Formula 2 represents the disturbance effect of each weight itself (i.e., the single weight disturbance factor).

[0105] For example, assuming that S is the minimum unit of weight clipping, for example, for a weight that can be clipped individually, S can be the single weight, and for weights that need to be clipped together, for example, multiple weights in a head, S can be the set of multiple weights, then the single weight perturbation factor can be:

[0106]

[0107] Where g represents the gradient, is the weight w i gradient.

[0108] The second term on the right side of Formula 2 represents the mutual influence between different structures (i.e., the weight-related disturbance factor mentioned above).

[0109] For example, the determination of the weight-related perturbation factor requires the calculation of the Hessian matrix. If the Hessian matrix is ​​calculated each time the weight-related perturbation factor is determined, a large amount of computing resources will be required. Since different structures play different roles, the entire Hessian matrix can be calculated according to the structural blocks, and the entire Hessian matrix can be divided into blocks according to different structural objects, such as Figure 4 That is, each structure object corresponds to a matrix block in the Hessian matrix, and different structure objects interact to form cross terms.

[0110] For example, H {kl} Represents the association matrix between structure object k and structure object 1:

[0111]

[0112] Where ΔW{k} Represents the overall weight change of the structure object k. For example, assuming that the structure object k has N neurons, then ΔW {k} It is an N-dimensional vector, and the value of each dimension represents the weight change value of each neuron (for example, from 0 to 1).

[0113] ΔW {k} H {kl} ΔW {l} Represents the interaction between structure object k and structure object l. The value range of k and l is the same, which is the number of structure objects after grouping. For example, if the neural network to be pruned is divided into 3 structure groups (i.e., the number of structure objects is 3), then the value range of k and l is (1, 2, 3).

[0114] For each matrix block, the following calculations can be performed:

[0115] ΔW {k} H {kl} ΔW {l} =N {k} N {l} u {kl} ρ {k} ρ {l} (5)

[0116] Among them, N {k} Represents the total number of weights of the structure object k, which can be directly obtained from the model parameters.

[0117] The matrix block consisting of structure object k and structure object 1 is calculated and only needs to be calculated once in advance.

[0118] in, Indicates W {k} H {kl} W {l} Find the average, W {k} H {kl} W {l} is a matrix containing multiple elements. For W {k} H {kl} W {l} Average the values ​​of the elements in .

[0119] ρ {k} Indicates the cropping rate of the structure object k.

[0120] It should be noted that the third term on the right side of the Taylor expansion formula of the above loss function has a very small effect on ΔL and can be ignored.

[0121] Therefore, the calculation of the entire approximate loss function change can be as follows:

[0122]

[0123] The above describes the method provided by this application. The following describes the device provided by this application:

[0124] See Figure 5 , is a structural diagram of a neural network model cutting device provided in an embodiment of the present application, such as Figure 5 As shown, the neural network model cutting device may include:

[0125] A division unit 510 is configured to divide the neural network model to be pruned into a plurality of structural groups according to structural characteristics of the neural network model to be pruned;

[0126] a determining unit 520 configured to determine a target cropping rate for cropping each of the plurality of structural groups based on a change in a loss function before and after cropping the neural network model to be cropped; wherein the change in the loss function is determined based on a first factor and a second factor, the first factor being used to characterize the importance of each weight in each structural group, and the second factor being used to characterize the interaction between different structural groups;

[0127] The cropping unit 530 is used to crop each structure group according to the target cropping rate to obtain a cropped neural network model.

[0128] In some embodiments, the neural network model to be pruned includes a Transform model;

[0129] The multiple structure groups include: a head in a self-attention structure, a fully connected layer in a feedforward neural network, and a fully connected layer constrained by a residual connection;

[0130] or,

[0131] The neural network model to be pruned includes a convolutional neural network model;

[0132] The division unit 510 divides the neural network model to be pruned into a plurality of structural groups, including:

[0133] The different network layers of the convolutional neural network model are divided into different structural groups; or, the different stages of the convolutional neural network model are divided into different structural groups.

[0134] In some embodiments, the determining unit 520 determines a target cropping rate for cropping each of the plurality of structure groups based on a change in the loss function before and after cropping the neural network model to be cropped, including:

[0135] Initializing a cropping rate population; wherein the cropping rate population includes a plurality of seeds, each seed including a cropping rate for cropping each of the plurality of structure groups;

[0136] According to the change of the loss function corresponding to each seed, the cropping rate population is iteratively updated until the number of iterations reaches a specified number of iterations, and the target cropping rate for cropping each of the multiple structure groups is determined according to the iteratively updated cropping rate population.

[0137] In some embodiments, the determining unit 520 iteratively updates the crop rate population according to the change of the loss function corresponding to each seed, including:

[0138] For any iteration, the various seeds in the current crop rate population are sorted according to the change in the loss function corresponding to the various seeds in the current crop rate population;

[0139] Based on the M seeds with the smallest loss function change in the current crop rate population, a new crop rate population is generated using crossover mutation; where M ≥ 1.

[0140] In some embodiments, the first factor is a single-weight perturbation factor, and the second factor is a weight-associated perturbation factor;

[0141] The change in the loss function before and after pruning the neural network model to be pruned is determined by:

[0142] Determining a change in the loss function before and after pruning the neural network model to be pruned based on the single weight perturbation factors in the multiple structural groups and the weight association perturbation factors between the multiple structural groups;

[0143] Among them, the single weight perturbation factor is used to characterize the performance loss of the neural network model before and after a single weight is pruned, and the weight association perturbation factor is used to characterize the performance loss of the neural network model before and after the associated weights are pruned.

[0144] In some embodiments, the weight association perturbation factors between the plurality of structure groups are determined by:

[0145] For the first structural grouping and the second structural grouping among the multiple structural groupings, the weight association disturbance factor between the first structural grouping and the second structural grouping is determined based on the number of weights in the first structural grouping, the number of weights in the second structural grouping, the clipping rate of the first structural grouping, the clipping rate of the second structural grouping, the first matrix block and the second matrix block; wherein the first matrix block is the matrix block corresponding to the first structural grouping in the Hessian matrix, and the second matrix block is the matrix block corresponding to the second structural grouping in the Hessian matrix.

[0146] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the neural network model pruning method described above.

[0147] See Figure 6 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 601 and a memory 602 storing machine-executable instructions. The processor 601 and the memory 602 may communicate via a system bus 603. Furthermore, by reading and executing the machine-executable instructions corresponding to the neural network model pruning logic in the memory 602, the processor 601 may execute the neural network model pruning method described above.

[0148] The memory 602 mentioned herein can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as a hard disk drive), solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0149] In some embodiments, a machine-readable storage medium is also provided. Figure 6 The memory 602 in the machine-readable storage medium stores machine-executable instructions. When the machine-executable instructions are executed by the processor, the neural network model pruning method described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0150] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0151] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A neural network model pruning method, characterized in that: include: Dividing the neural network model to be pruned into a plurality of structural groups according to structural characteristics of the neural network model to be pruned; The neural network model to be pruned includes a Transform model; The multiple structural groups include: a head in a self-attention structure, a fully connected layer in a feedforward neural network, and a fully connected layer constrained by residual connections; the Transform model includes a visual Transform model for image classification, the head in the self-attention structure is used to learn partial feature information of the image to be classified, including a feature tensor with multiple channels, each channel containing different information; each neuron in the fully connected layer in the feedforward neural network is used to fuse channel features; Based on the change in the loss function before and after pruning the neural network model to be pruned, a pruning rate search is performed with the structural grouping as the object, and a target pruning rate for pruning each of the multiple structural groupings is determined to reduce the performance loss of the pruned neural network model; wherein the change in the loss function is determined based on a first factor and a second factor, the first factor is used to characterize the importance of each weight in each structural grouping, and the second factor is used to characterize the interaction between different structural groupings; after pruning the multiple structural groups according to the target pruning rate, the computational amount of the pruned neural network model is consistent with the given computational amount target; According to the target cropping rate, each structure group is cropped separately to obtain a cropped neural network model.

2. The method according to claim 1, characterized in that Determining a target cropping rate for cropping each of the plurality of structure groups based on a change in a loss function before and after cropping the neural network model to be cropped includes: Initializing a cropping rate population; wherein the cropping rate population includes a plurality of seeds, each seed including a cropping rate for cropping each of the plurality of structure groups; According to the change of the loss function corresponding to each seed, the cropping rate population is iteratively updated until the number of iterations reaches a specified number of iterations, and the target cropping rate for cropping each of the multiple structure groups is determined according to the iteratively updated cropping rate population.

3. The method according to claim 2, characterized in that The iterative updating of the cropping rate population according to the change of the loss function corresponding to each seed includes: For any iteration, the various seeds in the current crop rate population are sorted according to the change in the loss function corresponding to the various seeds in the current crop rate population; Based on the M seeds with the smallest loss function change in the current crop rate population, a new crop rate population is generated using crossover mutation; where M ≥ 1.

4. The method according to claim 1, wherein The first factor is a single-weight perturbation factor, and the second factor is a weight-associated perturbation factor; The change in the loss function before and after pruning the neural network model to be pruned is determined by: Determining a change in the loss function before and after pruning the neural network model to be pruned based on the single weight perturbation factors in the multiple structural groups and the weight association perturbation factors between the multiple structural groups; Among them, the single weight perturbation factor is used to characterize the performance loss of the neural network model before and after a single weight is pruned, and the weight association perturbation factor is used to characterize the performance loss of the neural network model before and after the associated weights are pruned.

5. The method according to claim 4, characterized in that The weight association disturbance factors between the multiple structure groups are determined by: For the first structural grouping and the second structural grouping among the multiple structural groupings, the weight association disturbance factor between the first structural grouping and the second structural grouping is determined based on the number of weights in the first structural grouping, the number of weights in the second structural grouping, the clipping rate of the first structural grouping, the clipping rate of the second structural grouping, the first matrix block and the second matrix block; wherein the first matrix block is the matrix block corresponding to the first structural grouping in the Hessian matrix, and the second matrix block is the matrix block corresponding to the second structural grouping in the Hessian matrix.

6. A neural network model cutting device, characterized in that: include: A division unit, configured to divide the neural network model to be pruned into a plurality of structural groups according to structural characteristics of the neural network model to be pruned; The neural network model to be pruned includes a Transform model; The multiple structural groups include: a head in a self-attention structure, a fully connected layer in a feedforward neural network, and a fully connected layer constrained by residual connections; or, the neural network model to be pruned includes a convolutional neural network model; the dividing the neural network model to be pruned into multiple structural groups includes: dividing different network layers of the convolutional neural network model into different structural groups; or, dividing different stages of the convolutional neural network model into different structural groups; the Transform model includes a visual Transform model for image classification, the head in the self-attention structure is used to learn partial feature information of the image to be classified, including a feature tensor with multiple channels, each channel containing different information; each neuron in the fully connected layer in the feedforward neural network is used to fuse channel features; a determination unit configured to perform a cropping rate search based on a change in the loss function before and after pruning the neural network model to be pruned, taking the structural grouping as an object, and determine a target cropping rate for pruning each of the multiple structural groupings to reduce the performance loss of the pruned neural network model; wherein the change in the loss function is determined based on a first factor and a second factor, the first factor being used to characterize the importance of each weight in each structural grouping, and the second factor being used to characterize the interaction between different structural groupings; and after pruning the multiple structural groups based on the target cropping rate, the computational load of the pruned neural network model is consistent with a given computational load target; The cropping unit is used to crop each structure group according to the target cropping rate to obtain a cropped neural network model.

7. The device according to claim 6, characterized in that The determining unit determines a target cropping rate for cropping each of the plurality of structure groups based on a change in a loss function before and after cropping the neural network model to be cropped, including: Initializing a cropping rate population; wherein the cropping rate population includes a plurality of seeds, each seed including a cropping rate for cropping each of the plurality of structure groups; Iteratively updating the cropping rate population according to changes in the loss function corresponding to various seeds until the number of iterations reaches a specified number of iterations, and determining a target cropping rate for cropping each of the multiple structure groups according to the iteratively updated cropping rate population; The determining unit iteratively updates the cropping rate population according to the change of the loss function corresponding to each seed, including: For any iteration, the various seeds in the current crop rate population are sorted according to the change in the loss function corresponding to the various seeds in the current crop rate population; Based on the M seeds with the smallest loss function change in the current crop rate population, a new crop rate population is generated using crossover mutation; where M ≥ 1; And / or, the first factor is a single-weight perturbation factor, and the second factor is a weight-associated perturbation factor; The change in the loss function before and after pruning the neural network model to be pruned is determined by: Determining a change in the loss function before and after pruning the neural network model to be pruned based on the single weight perturbation factors in the multiple structural groups and the weight association perturbation factors between the multiple structural groups; The single weight perturbation factor is used to characterize the performance loss of the neural network model before and after a single weight is pruned, and the weight association perturbation factor is used to characterize the performance loss of the neural network model before and after the associated weights are pruned; The weight association disturbance factors between the multiple structural groups are determined by: For the first structural grouping and the second structural grouping among the multiple structural groupings, the weight association disturbance factor between the first structural grouping and the second structural grouping is determined based on the number of weights in the first structural grouping, the number of weights in the second structural grouping, the clipping rate of the first structural grouping, the clipping rate of the second structural grouping, the first matrix block and the second matrix block; wherein the first matrix block is the matrix block corresponding to the first structural grouping in the Hessian matrix, and the second matrix block is the matrix block corresponding to the second structural grouping in the Hessian matrix.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1 to 5.

9. A machine-readable storage medium, characterized in that The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images