A neural network model pruning method and device and a storage medium

Abstract

The embodiment of the application discloses a neural network model pruning method and device and a storage medium. The method comprises the following steps: obtaining a first neural network model corresponding to an original neural network model; wherein the first neural network model is a model obtained by inserting a compressor in a skip connection path of the original neural network model; the original neural network model comprises a main branch path and a skip connection path which are coupled with each other; the compressor is a network layer of the first neural network model and is used for decoupling the skip connection path and the main branch path from each other; taking the original neural network model as a benchmark, the first neural network model is subjected to model training to obtain a second neural network model; the second neural network model is subjected to network layer pruning to obtain a pruned neural network model corresponding to the original neural network model; and the pruned neural network model is used for data processing on multimedia data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and storage medium for pruning neural network models. Background Technology

[0002] Model pruning is the process of trimming an existing neural network model to effectively generate a smaller, more memory-efficient model with faster inference speed and less loss of accuracy.

[0003] Currently, the common method for model pruning is to perform inter-layer sensitivity analysis on the neural network model, and then set the pruning amount for each network layer in the neural network model based on the analysis results and the target pruning rate of computational cost, so as to perform channel pruning.

[0004] However, the UNet model, which is often used in low-level computer vision tasks, contains dense skip connections, i.e., dense skip paths. Skip paths are coupled with main paths. In actual inter-layer sensitivity analysis, the network layer at the starting position of the skip path belongs to both the skip path and the main path. Its sensitivity is affected by both the main path and the skip path, and it is often judged as having high sensitivity. In the end, the actual pruning rate is very low, which means that neither the skip path nor the main path can be effectively pruned, resulting in low pruning effectiveness of the neural network model. Summary of the Invention

[0005] This application provides a method, apparatus, and storage medium for pruning neural network models, which improves the effectiveness of pruning skip connections and main branches in neural network model pruning.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] This application provides a neural network model pruning method, including:

[0008] Obtain the first neural network model corresponding to the original neural network model; wherein, the first neural network model is a model obtained by inserting a compressor into the jump connection path of the original neural network model, the original neural network model includes mutually coupled main branch paths and the jump connection path, and the compressor serves as a network layer of the first neural network model to decouple the jump connection path from the main branch path;

[0009] Based on the original neural network model, the first neural network model is trained to obtain the second neural network model;

[0010] The second neural network model is pruned to obtain a pruned neural network model corresponding to the original neural network model; the pruned neural network model is used for data processing of multimedia data.

[0011] In the above method, obtaining the first neural network model corresponding to the original neural network model includes:

[0012] From the original neural network model, search for jump start network layers and jump end network layers that have jump connection relationships;

[0013] The connection path between the jump start network layer and the jump end network layer is defined as the jump connection path, and the compressor is inserted into the jump connection path to obtain the first neural network model.

[0014] In the above method, the step of training the first neural network model based on the original neural network model to obtain the second neural network model includes:

[0015] The training samples are processed using the first neural network model and the original neural network model respectively to obtain a first processing result corresponding to the first neural network model and a second processing result corresponding to the original neural network model.

[0016] Based on the first processing result and the second processing result, calculate the data loss function;

[0017] For each channel of the compressor in the first neural network model, a corresponding current weight norm penalty term is constructed based on the corresponding current weight, and a corresponding standard loss function is determined based on the data loss function and the corresponding current weight norm penalty term;

[0018] In the first neural network model, the weights of each channel of the compressor are adjusted based on the corresponding standard loss function, and the network layers different from the compressor are adjusted based on the data loss function to obtain the second neural network model.

[0019] In the above method, the step of pruning the network layers of the second neural network model to obtain the pruned neural network model corresponding to the original neural network model includes:

[0020] Inter-layer sensitivity analysis was performed on the second neural network model to obtain inter-layer sensitivity information;

[0021] Based on the target pruning rate and the inter-layer sensitivity information, the second neural network model is pruned to obtain the pruned neural network model.

[0022] In the above method, the inter-layer sensitivity information includes the network layer sensitivity corresponding to each network layer in the second neural network model. The step of performing inter-layer sensitivity analysis on the second neural network model to obtain the inter-layer sensitivity information includes:

[0023] By using different pruning rates, channel pruning is performed on the target network layer in the second neural network model to obtain multiple pruning models corresponding to the target network layer; wherein, the target network layer is any network layer in the second neural network model, and each pruning model is a model obtained by pruning the target network layer using a pruning rate;

[0024] For each of the multiple pruning models corresponding to the target network layer, model training is performed and the accuracy of the trained model is tested to determine the corresponding fine-tuning accuracy.

[0025] The pruning rate and fine-tuning accuracy of each pruning model among the multiple pruning models corresponding to the target network layer are determined as the network layer sensitivity corresponding to the target network layer.

[0026] Continue to determine the corresponding network layer sensitivity for network layers in the second neural network model that are different from the target network layer, until the network layer sensitivity corresponding to each network layer in the second neural network model is obtained.

[0027] In the above method, the step of pruning the network layers of the second neural network model based on the target pruning rate and the inter-layer sensitivity information to obtain the pruned neural network model includes:

[0028] Based on the target pruning rate and the inter-layer sensitivity information, the pruning rate corresponding to each network layer in the second neural network model is determined;

[0029] For each network layer in the second neural network model, the number of channels to be pruned is determined according to the corresponding pruning rate;

[0030] For the second neural network model, in each network layer, the weight norm corresponding to each channel in the network layer is obtained, and the channels are selected in ascending order of the corresponding weight norm until the number of selected channels reaches the corresponding number of channels to be pruned. The selected channels are then pruned to obtain the pruned neural network model.

[0031] In the above method, after pruning the network layers of the second neural network model to obtain the pruned neural network model corresponding to the original neural network model, the method further includes:

[0032] From the pruned neural network model, at least one network layer with a convolution kernel size larger than a preset size is selected, and each network layer in the at least one network layer is structurally reparameterized to obtain at least one structural reconstruction block corresponding to the at least one network layer.

[0033] In the pruned neural network model, at least one network layer is replaced with at least one structural reconstruction block to obtain a third neural network model;

[0034] The third neural network model is trained, and each trained reconstruction block in at least one trained reconstruction block in the trained neural network model is converted into a network layer to obtain the target neural network model; wherein each trained reconstruction block in the at least one trained reconstruction block is trained by one structural reconstruction block in the at least one structural reconstruction block.

[0035] In the above method, the step of performing structural reparameterization on each of the at least one network layer to obtain at least one structural reconstruction block corresponding one-to-one with the at least one network layer includes:

[0036] For the first network layer with a convolution kernel size of K×K in the at least one network layer, a first branch, a second branch, and a third branch are generated; wherein, the first network layer is any one of the at least one network layers, K is a natural number greater than 1, the first branch is a network layer with a convolution kernel size of 1×1, the second branch is composed of a convolution kernel with a convolution kernel size of 1×1 and a network layer with a convolution kernel size of K×K connected in series, and the third branch is composed of a convolution kernel with a convolution kernel size of 1×1 and an average pooling layer connected in series;

[0037] The first network layer, the first branch, the second branch, and the third branch are combined to obtain the structural reconstruction block corresponding to the first network layer.

[0038] This application provides a neural network model pruning device, including:

[0039] An acquisition module is used to acquire a first neural network model corresponding to the original neural network model; wherein, the first neural network model is a model obtained by inserting a compressor into the jump connection path of the original neural network model, the original neural network model includes mutually coupled main branch paths and the jump connection path, and the compressor serves as a network layer of the first neural network model to decouple the jump connection path from the main branch path;

[0040] The training module is used to train the first neural network model based on the original neural network model to obtain a second neural network model.

[0041] The pruning module is used to prune the network layers of the second neural network model to obtain a pruned neural network model corresponding to the original neural network model; the pruned neural network model is used for data processing of multimedia data.

[0042] This application provides a neural network model pruning device, including: a processor, a memory, and a communication bus;

[0043] The communication bus is used to realize the communication connection between the processor and the memory;

[0044] The processor is used to execute one or more computer programs stored in the memory to implement the above-described neural network model pruning method.

[0045] This application provides a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the above-described neural network model pruning method.

[0046] This application provides a method, apparatus, and storage medium for pruning a neural network model. The method includes: obtaining a first neural network model corresponding to an original neural network model; wherein the first neural network model is a model obtained by inserting a compressor into the jump connections of the original neural network model, the original neural network model including mutually coupled main branches and jump connections, and the compressor serving as a network layer of the first neural network model to decouple the jump connections from the main branches; training the first neural network model based on the original neural network model to obtain a second neural network model; pruning the network layers of the second neural network model to obtain a pruned neural network model corresponding to the original neural network model; the pruned neural network model is used for data processing of multimedia data. The technical solution provided by this application provides that the introduction of a compressor into the jump connections decouples the main branches from the jump connections, allowing the jump connections and main branches to be pruned separately, thereby improving the effectiveness of pruning the jump connections and main branches. Attached Figure Description

[0047] Figure 1 A flowchart illustrating a neural network model pruning method provided in an embodiment of this application;

[0048] Figure 2 A schematic diagram of the structure of an exemplary first neural network model provided in an embodiment of this application;

[0049] Figure 3 A schematic diagram of an exemplary structure of a diverse branch provided for an embodiment of this application;

[0050] Figure 4A schematic diagram of pruning an exemplary original neural network model provided for an embodiment of this application;

[0051] Figure 5 A schematic diagram of the structure of a neural network model pruning device provided in this application embodiment. Figure 1 ;

[0052] Figure 6 A schematic diagram of the structure of a neural network model pruning device provided in this application embodiment. Figure 2 . Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The technical solutions of this application and how they solve the aforementioned technical problems will be described in detail below through embodiments and in conjunction with the accompanying drawings. The embodiments below can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0055] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0056] This application provides a method for pruning neural network models, implemented using a neural network model pruning device. The neural network model pruning device can be any type of electronic device, and this application does not limit its application. Figure 1 This is a flowchart illustrating a neural network model pruning method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the neural network model pruning method includes:

[0057] S101. Obtain the first neural network model corresponding to the original neural network model; wherein, the first neural network model is a model obtained by inserting a compressor into the jump connection path of the original neural network model. The original neural network model includes mutually coupled main branch paths and jump connection paths. The compressor serves as a network layer of the first neural network model and is used to decouple the jump connection path from the main branch path.

[0058] In the embodiments of this application, the neural network model pruning device can obtain a first neural network model corresponding to the original neural network model; wherein, the first neural network model is a model obtained by inserting a compressor into the jump connection path of the original neural network model, the original neural network model includes mutually coupled main branch paths and jump connection paths, and the compressor serves as a network layer of the first neural network model to decouple the jump connection path from the main branch path.

[0059] It should be noted that, in the embodiments of this application, the original neural network model can be any type of model that requires pruning, and it includes mutually coupled main branches and skip connections. Specifically, the original neural network model can be a UNET neural network model, which has dense skip connections, i.e., includes dense skip connections. Of course, the original neural network model is also set according to actual needs and application scenarios, and this embodiment of the application does not limit it.

[0060] It is understood that, in the embodiments of this application, considering the great redundancy of the jump connection part in the model, a compressor can be inserted into the jump connection of the original neural network model to decouple the jump connection from the main branch path. This can avoid the situation where the network layer at the starting position of the jump connection is affected by both the main branch path and the jump connection during model pruning, resulting in a low pruning rate, thus ensuring the effectiveness of subsequent model pruning.

[0061] In an embodiment of this application, the neural network model pruning device obtains a first neural network model corresponding to the original neural network model, including: searching for jump start network layers and jump end network layers with jump connections in the original neural network model; determining the connection path between the jump start network layer and the jump end network layer as a jump connection path; and inserting a compressor into the jump connection path to obtain the first neural network model.

[0062] It should be noted that, in the embodiments of this application, the jump connection in the original neural network model can be one or more, and the neural network model pruning device can insert a compressor for each jump connection, thereby achieving decoupling of each jump connection from the main branch path.

[0063] It is understood that, in the embodiments of this application, the first neural network model is structurally the same as the original neural network model, except that a compressor is inserted in the jump connection path.

[0064] Figure 2 This is a schematic diagram illustrating the structure of an exemplary first neural network model provided in an embodiment of this application. In the embodiments of this application, the original neural network model can be a UNET neural network model, and by inserting a compressor into its skip connections, the following can be obtained: Figure 2 The first neural network model shown includes convolutional layers, upsampling layers, downsampling layers, and a compressor, all of which are network layers in the first neural network model. Figure 2As shown, the solid lines represent the main branch paths, and the dashed lines represent each of the four jump paths, namely jump path 1, jump path 2, jump path 3, and jump path 4. In each of these four jump paths, a compressor is inserted. Thus, subsequent pruning of the jump paths is actually reflected in the pruning of the compressors within the jump paths, so that the jump paths can be separated from the main branch paths for pruning.

[0065] It should be noted that, in the embodiments of this application, the compressor can specifically be a 1×1 convolutional kernel with an equal number of input and output channels, which, when introduced into the model, has no impact on the model output. Of course, the compressor can also be a lightweight fully connected layer, or other forms of network layers, to reduce the additional computational load introduced by the compressor. The specific form of the compressor is not limited in the embodiments of this application. Furthermore, the compressor, as a network layer in the first neural network model, is used for pruning subsequent network layers participating in the model.

[0066] S102. Using the original neural network model as a benchmark, train the first neural network model to obtain the second neural network model.

[0067] In the embodiments of this application, after obtaining the first neural network model, the neural network model pruning device trains the first neural network model based on the original neural network model to obtain the second neural network model.

[0068] It should be noted that, in the embodiments of this application, in order to embed the sensitivity of the jump connection into the compressor, after obtaining the first neural network model, the neural network model pruning device needs to train the first neural network model based on the original neural network model to obtain the second neural network model.

[0069] In the embodiments of this application, the neural network model pruning device trains a first neural network model based on the original neural network model, including: processing training samples using the first neural network model and the original neural network model respectively to obtain a first processing result corresponding to the first neural network model and a second processing result corresponding to the original neural network model; calculating a data loss function based on the first processing result and the second processing result; constructing a corresponding current weight norm penalty term for each channel of the compressor in the first neural network model based on the corresponding current weight, and determining a corresponding standard loss function based on the data loss function and the corresponding current weight norm penalty term; adjusting the weights of each channel of the compressor in the first neural network model based on the corresponding standard loss function, and adjusting network layers different from the compressor based on the data loss function to obtain a second neural network model.

[0070] It should be noted that in the embodiments of this application, there can be multiple training samples. This embodiment of the application does not limit the number of training samples. Each time the above process is executed with a training sample, it is considered to complete one training cycle.

[0071] It should be noted that in the embodiments of this application, since the compressor inevitably carries information from the model during training, a "ResRep" training process can be introduced to minimize the pruning loss of the compressor. This introduces a penalty term for the weight norm of the compressor. That is, when adjusting the weights of each channel in the compressor during a training cycle, not only is the loss generated by the model processing data (i.e., the data loss function) applied, but also the current weight norm penalty term corresponding to the channel is combined. The neural network model pruning device can sum the data loss function and the current weight norm penalty term corresponding to the channel as the standard loss function for that channel. After several training cycles, several channels to be pruned can be selected from the compressor based on the magnitude of the channel's weight norm. When determining the standard loss function for these channels, the data loss function is directly set to zero, reducing the antagonism between the two loss functions and making the weights of the selected channels closer to 0.

[0072] It should be noted that, in the embodiments of this application, the neural network model pruning device uses the original neural network model as a benchmark to train the first neural network model. This includes not only the training of the compressor mentioned above, but also the training of other network layers in the first neural network model that are different from the compressor. For these network layers, the data loss function can be directly used to adjust the channel weights, which will not be elaborated here.

[0073] S103. Perform network layer pruning on the second neural network model to obtain the pruned neural network model corresponding to the original neural network model; the pruned neural network model is used for data processing of multimedia data.

[0074] In the embodiments of this application, the neural network model pruning device can perform network layer pruning on the second neural network model after obtaining the second neural network model, so as to obtain the pruned neural network model corresponding to the original neural network model.

[0075] In the embodiments of this application, the neural network model pruning device performs network layer pruning on the second neural network model to obtain a pruned neural network model corresponding to the original neural network model, including: performing inter-layer sensitivity analysis on the second neural network model to obtain inter-layer sensitivity information; and performing network layer pruning on the second neural network model based on the target pruning rate and inter-layer sensitivity information to obtain a pruned neural network model.

[0076] In the embodiments of this application, the inter-layer sensitivity information includes the network layer sensitivity corresponding to each network layer in the second neural network model. The neural network model pruning device performs inter-layer sensitivity analysis on the second neural network model to obtain the inter-layer sensitivity information, including: using different pruning rates to perform channel pruning on the target network layer in the second neural network model to obtain multiple pruning models corresponding to the target network layer; wherein, the target network layer is any network layer in the second neural network model, and each pruning model is a model obtained by pruning the target network layer using a pruning rate; for each of the multiple pruning models corresponding to the target network layer, the model is trained and the model accuracy of the trained model is tested to determine the corresponding fine-tuning accuracy; the pruning rate and fine-tuning accuracy corresponding to each of the multiple pruning models corresponding to the target network layer are determined as the network layer sensitivity corresponding to the target network layer; the network layer sensitivity is determined for network layers in the second neural network model that are different from the target network layer, until the network layer sensitivity corresponding to each network layer in the second neural network model is obtained.

[0077] It should be noted that, in the embodiments of this application, the neural network model pruning device can determine the sensitivity of each network layer for the second neural network model layer by layer. For any network layer in the second neural network model, i.e., the target network layer, the neural network model pruning device can perform channel pruning with pruning rates of 0.1, 0.2, 0.3, ..., 0.9, while keeping other network layers in the second neural network unchanged. In this way, each model obtained is a pruned model corresponding to the target network layer. Then, the neural network model pruning device can introduce one or more cycles of fine-tuning for each pruned model, i.e., perform one or more cycles of model training. This can make the final analysis results more accurate. Finally, the accuracy is tested on the test set and used as the fine-tuning accuracy corresponding to the pruned model. The pruning rate and fine-tuning accuracy corresponding to each pruned model are used as the sensitivity of the network layer corresponding to the target network layer.

[0078] It should be noted that, in the embodiments of this application, the method used to determine the network layer sensitivity corresponding to the network layer that is different from the target network layer in the second neural network model by model pruning is the same as that used to determine the network layer sensitivity corresponding to the target network layer, and will not be repeated here.

[0079] In the embodiments of this application, the neural network model pruning device performs network layer pruning on the second neural network model based on the target pruning rate and inter-layer sensitivity information to obtain a pruned neural network model. This includes: determining the pruning rate corresponding to each network layer in the second neural network model based on the target pruning rate and inter-layer sensitivity information; determining the corresponding number of channels to be pruned for each network layer in the second neural network model according to the corresponding pruning rate; and for the second neural network model, selecting channels in each network layer according to the norm of the corresponding weights in ascending order until the number of selected channels reaches the corresponding number of channel prunings, then pruning the selected channels to obtain the pruned neural network model.

[0080] It should be noted that, in the embodiments of this application, the target pruning rate is a pruning rate set in advance according to the actual pruning requirements. The neural network model pruning device can automatically determine the pruning rate corresponding to each network layer in the second neural network model according to the inter-layer sensitivity information and the target pruning rate. Furthermore, for each network layer, the corresponding number of channels to be pruned is determined according to the corresponding pruning rate, and that number of channels is pruned.

[0081] It should be noted that, in the embodiments of this application, when performing channel pruning on each network layer in the second neural network model, the neural network model pruning device can prune channels with smaller norms corresponding to the weights. The number of channels pruned is the number of channels pruned for the network layer.

[0082] It should be noted that, in the embodiments of this application, the pruned neural network model can be used to process multimedia data. Specific multimedia data can be text, images, audio, or other types of data, and this application does not limit this. For example, the original neural network model is a neural network model with object detection capabilities; correspondingly, the pruned neural network model is essentially a lightweight neural network model with object detection capabilities. Therefore, the pruned neural network model can be used to detect objects in a target image, thereby identifying the detected objects from the target image. Furthermore, the original neural network model can also be a neural network model with target tracking capabilities; correspondingly, the pruned neural network model is essentially a lightweight neural network model with target tracking capabilities. Therefore, the pruned neural network model can be used to track target objects in a target video, thereby identifying the target objects in each frame of the target video. The type of multimedia data processed by the pruned neural network model matches the functions possessed by the pruned neural network model, and can be specifically set according to actual needs and application scenarios; this application does not limit this.

[0083] In the embodiments of this application, the neural network model pruning device performs network layer pruning on the second neural network model to obtain the pruned neural network model corresponding to the original neural network model. After that, the pruned neural network model can be fine-tuned to obtain the target neural network model.

[0084] In the embodiments of this application, after the neural network model pruning device prunes the network layers of the second neural network model to obtain the pruned neural network model corresponding to the original neural network model, it can also perform the following steps: from the pruned neural network model, select at least one network layer with a convolution kernel size larger than a preset size, and perform structural reparameterization on each of the at least one network layer to obtain at least one structural reconstruction block corresponding to the at least one network layer; replace at least one network layer in the pruned neural network model with at least one structural reconstruction block to obtain a third neural network model; train the third neural network model, and convert each of the at least one trained reconstruction block in the trained neural network model into a network layer to obtain the target neural network model; wherein each of the at least one trained reconstruction block is trained by one of the at least one structural reconstruction blocks.

[0085] It should be noted that, in the embodiments of this application, in order to improve the model's representation ability during fine-tuning without affecting the final number of model parameters, the neural network model pruning device can introduce structural reparameterization of the model during the fine-tuning training process.

[0086] In embodiments of this application, a neural network model pruning device performs structural reparameterization on each of at least one network layer to obtain at least one structural reconstruction block corresponding to at least one network layer. This includes: generating a first branch, a second branch, and a third branch for a first network layer with a convolution kernel size of K×K; wherein the first network layer is any one of the at least one network layers, K is a natural number greater than 1, the first branch is a network layer with a convolution kernel size of 1×1, the second branch is composed of a convolution kernel with a convolution kernel size of 1×1 and a network layer with a convolution kernel size of K×K connected in series, and the third branch is composed of a convolution kernel with a convolution kernel size of 1×1 and an average pooling layer connected in series; combining the first network layer, the first branch, the second branch, and the third branch yields the structural reconstruction block corresponding to the first network layer.

[0087] It should be noted that, in the embodiments of this application, the neural network model pruning device can find network layers in the pruned neural network model with a convolutional kernel size greater than 1×1 and expand them into Diverse Branch Blocks (DBBs). After fine-tuning, each DBB module can be merged into the original convolutional kernel of the model, thereby obtaining a lightweight inference model that is comparable to the model before pruning and transformation, i.e., the target neural network model, without any loss of accuracy in this process.

[0088] Figure 3 This is a schematic diagram illustrating an exemplary structure of a diverse branch, provided as an embodiment of this application. For example... Figure 3 As shown in the embodiments of this application, the pruned neural network model includes a network layer with a convolution kernel size of K×K, i.e. Figure 3 The part shown within the dashed box is where K is a natural number greater than 1. The neural network model pruning device can add three branches to this network layer. The first branch is a network layer with a convolution kernel size of 1×1. The second branch is composed of a network layer with a convolution kernel size of 1×1 and a network layer with a convolution kernel size of K×K connected in series. The third branch is composed of a network layer with a convolution kernel size of 1×1 and an average pooling layer connected in series. The inputs of the first, second, and third branches are all connected to the input of this network layer. The first, second, and third branches, as well as the output of this network layer, are connected to the same adder. Thus, the output of this network layer is finally added together with the outputs of the first, second, and third branches through the adder and output.

[0089] It should be noted that there are multiple options for the structure reparameterization method in the embodiments of this application. For example, various forms of single or multiple branches can be added, or a structure reparameterization method suitable for the underlying computer vision task and UNet-like model can be selected. The embodiments of this application do not limit this.

[0090] Figure 4 This is a schematic diagram of pruning an exemplary original neural network model provided in an embodiment of this application. Figure 4As shown in the embodiments of this application, the neural network model pruning device can sequentially perform the following steps on the original neural network model: model conversion, i.e., obtaining and training the first neural network model corresponding to the original neural network model to obtain the second neural network model, which is actually the object of subsequent model pruning; inter-layer sensitivity analysis, mainly after model conversion, performing sensitivity analysis on the network layers in the obtained second neural network model to obtain inter-layer sensitivity information, wherein the analysis process involves fine-tuning, i.e., after pruning a network layer in the second neural network model with different pruning rates, training the obtained pruned models first, and then testing the model accuracy; channel pruning, mainly after inter-layer sensitivity analysis, performing network layer pruning on the second neural network model based on the obtained inter-layer sensitivity information and target pruning rate to obtain the pruned neural network model; fine-tuning, mainly after obtaining the pruned neural network model, introducing structural reparameterization to fine-tune the pruned neural network model.

[0091] Understandably, in the embodiments of this application, firstly, the introduction of a compressor in the skip connection decouples the main branch path from the skip connection. Due to the redundancy of the skip connection, this operation can improve the pruning rate of the skip connection and reduce computational and memory overhead. Secondly, the inter-layer sensitivity analysis introduced for fine-tuning is more accurate, and many cases will appear where high pruning rates are more effective than low pruning rates, which is beneficial for achieving a higher pruning rate than before. Thirdly, the introduction of structural reparameterization in the fine-tuning stage can increase the representational ability during the training stage without increasing the computational load during the final deployment of the model. In the embodiments of this application, the original neural network model can be a UNet model. After pruning the original UNet model using the model pruning method provided in this application, the performance of the obtained model on a specific dataset is verified. Compared with the original UNet model, the computational load is reduced by 50.2%, the number of parameters is reduced by 78.6%, and the peak signal-to-noise ratio (PSNR) accuracy decreases by 0.255%.

[0092] This application provides a model pruning method, comprising: obtaining a first neural network model corresponding to an original neural network model; wherein the first neural network model is a model obtained by inserting a compressor into the jump connections of the original neural network model, the original neural network model including mutually coupled main branches and jump connections, the compressor serving as a network layer of the first neural network model, used to decouple the jump connections from the main branches; training the first neural network model based on the original neural network model to obtain a second neural network model; pruning the network layers of the second neural network model to obtain a pruned neural network model corresponding to the original neural network model; the pruned neural network model being used for data processing of multimedia data. The model pruning method provided in this application, by introducing a compressor into the jump connections, decouples the main branches from the jump connections, allowing the jump connections and main branches to be pruned separately, thereby improving the effectiveness of pruning the jump connections and main branches.

[0093] This application provides a neural network model pruning device. Figure 5 A schematic diagram of the structure of a neural network model pruning device provided in this application embodiment. Figure 1 .like Figure 5 As shown in the embodiments of this application, the neural network model pruning device includes:

[0094] The acquisition module 501 is used to acquire the first neural network model corresponding to the original neural network model; wherein, the first neural network model is a model obtained by inserting a compressor into the jump connection path of the original neural network model. The original neural network model includes mutually coupled main branch paths and jump connection paths. The compressor serves as a network layer of the first neural network model and is used to decouple the jump connection path from the main branch path.

[0095] The training module 502 is used to train the first neural network model based on the original neural network model to obtain the second neural network model.

[0096] The pruning module 503 is used to prune the network layers of the second neural network model to obtain the pruned neural network model corresponding to the original neural network model; the pruned neural network model is used for data processing of multimedia data.

[0097] In one embodiment of this application, the acquisition module 501 is further configured to search for jump start network layers and jump end network layers with jump connection relationships in the original neural network model; determine the connection path between the jump start network layer and the jump end network layer as a jump connection path; and insert a compressor into the jump connection path to obtain the first neural network model.

[0098] In one embodiment of this application, the training module 502 is further configured to process the training samples using a first neural network model and an original neural network model respectively, to obtain a first processing result corresponding to the first neural network model and a second processing result corresponding to the original neural network model; calculate a data loss function based on the first processing result and the second processing result; construct a corresponding current weight norm penalty term for each channel of the compressor in the first neural network model based on the corresponding current weight, and determine the corresponding standard loss function based on the data loss function and the corresponding current weight norm penalty term; adjust the weights of each channel of the compressor in the first neural network model based on the corresponding standard loss function, and adjust the network layers different from the compressor based on the data loss function to obtain a second neural network model.

[0099] In one embodiment of this application, the pruning module 503 is further configured to perform inter-layer sensitivity analysis on the second neural network model to obtain inter-layer sensitivity information; and to perform network layer pruning on the second neural network model based on the target pruning rate and inter-layer sensitivity information to obtain a pruned neural network model.

[0100] In one embodiment of this application, the inter-layer sensitivity information includes the network layer sensitivity corresponding to each network layer in the second neural network model. The pruning module 503 is further used to perform channel pruning on the target network layer in the second neural network model using different pruning rates to obtain multiple pruning models corresponding to the target network layer. The target network layer is any network layer in the second neural network model, and each pruning model is a model obtained by pruning the target network layer using a pruning rate. For each pruning model among the multiple pruning models corresponding to the target network layer, model training is performed and the model accuracy of the trained model is tested to determine the corresponding fine-tuning accuracy. The pruning rate and fine-tuning accuracy corresponding to each pruning model among the multiple pruning models corresponding to the target network layer are determined as the network layer sensitivity corresponding to the target network layer. The network layer sensitivity is then determined for network layers in the second neural network model that are different from the target network layer, until the network layer sensitivity corresponding to each network layer in the second neural network model is obtained.

[0101] In one embodiment of this application, the pruning module 503 is further configured to determine the pruning rate corresponding to each network layer in the second neural network model based on the target pruning rate and inter-layer sensitivity information; for each network layer in the second neural network model, determine the corresponding number of channels to be pruned according to the corresponding pruning rate; for the second neural network model, in each network layer, channels are selected in ascending order of the norm of the corresponding weights until the number of selected channels reaches the corresponding number of channels to be pruned, and the selected channels are pruned to obtain the pruned neural network model.

[0102] In one embodiment of this application, the training module 502 is further configured to: select at least one network layer with a convolution kernel size larger than a preset size from the pruned neural network model; perform structural reparameterization on each of the at least one network layer to obtain at least one structural reconstruction block corresponding to the at least one network layer; replace at least one network layer in the pruned neural network model with at least one structural reconstruction block to obtain a third neural network model; train the third neural network model; and convert each of the at least one trained reconstruction block in the trained neural network model into a network layer to obtain a target neural network model; wherein each of the at least one trained reconstruction block is trained from one of the at least one structural reconstruction blocks.

[0103] In one embodiment of this application, the training module 502 is further configured to generate a first branch, a second branch, and a third branch for a first network layer with a convolution kernel size of K×K in at least one network layer; wherein the first network layer is any one of the at least one network layers, K is a natural number greater than 1, the first branch is a network layer with a convolution kernel size of 1×1, the second branch is composed of a convolution kernel with a convolution kernel size of 1×1 and a network layer with a convolution kernel size of K×K connected in series, and the third branch is composed of a convolution kernel with a convolution kernel size of 1×1 and an average pooling layer connected in series; the first network layer, the first branch, the second branch, and the third branch are combined to obtain the structural reconstruction block corresponding to the first network layer.

[0104] Figure 6 A schematic diagram of the structure of a neural network model pruning device provided in this application embodiment. Figure 2 .like Figure 6 As shown in the embodiments of this application, the neural network model pruning device includes: a processor 601, a memory 602, and a communication bus 603;

[0105] Communication bus 603 is used to realize the communication connection between processor 601 and memory 602;

[0106] The processor 601 is used to execute one or more computer programs stored in the memory 602 to implement the neural network model pruning method described above.

[0107] This application provides a neural network model pruning device. A compressor is inserted into the jump connections of an original neural network model to obtain a first neural network model. The original neural network model includes mutually coupled main branches and jump connections. The compressor serves as a network layer in the first neural network model, decoupling the jump connections from the main branches. Based on the original neural network model, the first neural network model is trained to obtain a second neural network model. Network layer pruning is performed on the second neural network model to obtain a pruned neural network model corresponding to the original neural network model. The pruned neural network model is used for data processing of multimedia data. The neural network model pruning device provided in this application provides a method to decouple the main branches from the jump connections by introducing a compressor into the jump connections, allowing for separate pruning of the jump connections and main branches, thereby improving the effectiveness of pruning the jump connections and main branches while reducing computational load and memory consumption.

[0108] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the aforementioned neural network model pruning method. The computer-readable storage medium can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or it can be a device including one or any combination of the above-mentioned memories, such as a mobile phone, computer, tablet device, personal digital assistant, etc.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0110] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for pruning a neural network model, characterized in that, include: Obtain the first neural network model corresponding to the original neural network model; wherein, the first neural network model is a model obtained by inserting a compressor into the jump connection path of the original neural network model, the original neural network model includes mutually coupled main branch paths and the jump connection path, and the compressor serves as a network layer of the first neural network model to decouple the jump connection path from the main branch path; Based on the original neural network model, the first neural network model is trained to obtain the second neural network model; The second neural network model is pruned to obtain a pruned neural network model corresponding to the original neural network model; the original neural network model and the pruned neural network model are used to process multimedia data, which includes one or more of text, images, and audio. The step of training the first neural network model based on the original neural network model to obtain the second neural network model includes: The training samples are processed using the first neural network model and the original neural network model respectively to obtain a first processing result corresponding to the first neural network model and a second processing result corresponding to the original neural network model. Based on the first processing result and the second processing result, calculate the data loss function; For each channel of the compressor in the first neural network model, a corresponding current weight norm penalty term is constructed based on the corresponding current weight, and a corresponding standard loss function is determined based on the data loss function and the corresponding current weight norm penalty term; In the first neural network model, the weights of each channel of the compressor are adjusted based on the corresponding standard loss function, and the network layers different from the compressor are adjusted based on the data loss function to obtain the second neural network model.

2. The method according to claim 1, characterized in that, The step of obtaining the first neural network model corresponding to the original neural network model includes: From the original neural network model, search for jump start network layers and jump end network layers that have jump connection relationships; The connection path between the jump start network layer and the jump end network layer is defined as the jump connection path, and the compressor is inserted into the jump connection path to obtain the first neural network model.

3. The method according to claim 1, characterized in that, The step of pruning the network layers of the second neural network model to obtain the pruned neural network model corresponding to the original neural network model includes: Inter-layer sensitivity analysis was performed on the second neural network model to obtain inter-layer sensitivity information; Based on the target pruning rate and the inter-layer sensitivity information, the second neural network model is pruned to obtain the pruned neural network model.

4. The method according to claim 3, characterized in that, The inter-layer sensitivity information includes the network layer sensitivity corresponding to each network layer in the second neural network model; the step of performing inter-layer sensitivity analysis on the second neural network model to obtain inter-layer sensitivity information includes: By using different pruning rates, channel pruning is performed on the target network layer in the second neural network model to obtain multiple pruning models corresponding to the target network layer; wherein, the target network layer is any network layer in the second neural network model, and each pruning model is a model obtained by pruning the target network layer using a pruning rate; For each of the multiple pruning models corresponding to the target network layer, model training is performed and the accuracy of the trained model is tested to determine the corresponding fine-tuning accuracy. The pruning rate and fine-tuning accuracy of each pruning model among the multiple pruning models corresponding to the target network layer are determined as the network layer sensitivity corresponding to the target network layer. Continue to determine the corresponding network layer sensitivity for network layers in the second neural network model that are different from the target network layer, until the network layer sensitivity corresponding to each network layer in the second neural network model is obtained.

5. The method according to claim 3, characterized in that, The step of pruning network layers in the second neural network model based on the target pruning rate and the inter-layer sensitivity information to obtain the pruned neural network model includes: Based on the target pruning rate and the inter-layer sensitivity information, the pruning rate corresponding to each network layer in the second neural network model is determined; For each network layer in the second neural network model, the number of channels to be pruned is determined according to the corresponding pruning rate; For the second neural network model, in each network layer, channels are selected in ascending order of the norm of the corresponding weights until the number of selected channels reaches the corresponding number of channels to be pruned. The selected channels are then pruned to obtain the pruned neural network model.

6. The method according to any one of claims 1-5, characterized in that, After pruning the network layers of the second neural network model to obtain the pruned neural network model corresponding to the original neural network model, the method further includes: From the pruned neural network model, at least one network layer with a convolution kernel size larger than a preset size is selected, and each network layer in the at least one network layer is structurally reparameterized to obtain at least one structural reconstruction block corresponding to the at least one network layer. In the pruned neural network model, at least one network layer is replaced with at least one structural reconstruction block to obtain a third neural network model; The third neural network model is trained, and each trained reconstruction block in at least one trained reconstruction block in the trained neural network model is converted into a network layer to obtain the target neural network model; wherein each trained reconstruction block in the at least one trained reconstruction block is trained by one structural reconstruction block in the at least one structural reconstruction block.

7. The method according to claim 6, characterized in that, The step of reparameterizing the structure of each of the at least one network layer to obtain at least one structure reconstruction block corresponding one-to-one with the at least one network layer includes: For the first network layer with a convolution kernel size of K×K in the at least one network layer, a first branch, a second branch, and a third branch are generated; wherein, the first network layer is any one of the at least one network layers, K is a natural number greater than 1, the first branch is a network layer with a convolution kernel size of 1×1, the second branch is composed of a convolution kernel with a convolution kernel size of 1×1 and a network layer with a convolution kernel size of K×K connected in series, and the third branch is composed of a convolution kernel with a convolution kernel size of 1×1 and an average pooling layer connected in series; The first network layer, the first branch, the second branch, and the third branch are combined to obtain the structural reconstruction block corresponding to the first network layer.

8. A neural network model pruning device, characterized in that, include: An acquisition module is used to acquire a first neural network model corresponding to the original neural network model; wherein, the first neural network model is a model obtained by inserting a compressor into the jump connection path of the original neural network model, the original neural network model includes mutually coupled main branch paths and the jump connection path, and the compressor serves as a network layer of the first neural network model to decouple the jump connection path from the main branch path; The training module is used to train the first neural network model based on the original neural network model to obtain the second neural network model. The pruning module is used to prune the network layers of the second neural network model to obtain a pruned neural network model corresponding to the original neural network model; the original neural network model and the pruned neural network model are used to process multimedia data, which includes one or more of text, images, and audio. The training module is used to process training samples using the first neural network model and the original neural network model respectively, to obtain a first processing result corresponding to the first neural network model and a second processing result corresponding to the original neural network model; to calculate a data loss function based on the first processing result and the second processing result; to construct a corresponding current weight norm penalty term for each channel of the compressor in the first neural network model based on the corresponding current weight, and to determine a corresponding standard loss function based on the data loss function and the corresponding current weight norm penalty term; to adjust the weights of each channel of the compressor in the first neural network model based on the corresponding standard loss function, and to adjust network layers different from the compressor based on the data loss function, to obtain the second neural network model.

9. A neural network model pruning device, characterized in that, include: Processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is configured to execute one or more computer programs stored in the memory to implement the neural network model pruning method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the neural network model pruning method as described in any one of claims 1-7.

Images