Neural Network Pruning Methods

Abstract

The present application relates to a method for pruning a neural network model. The method comprises: obtaining labeled training data and high-imitation data; inputting the training data and high-imitation data into the neural network model to obtain a plurality of training features and a plurality of high-imitation features respectively; each operation operator on each operation layer is assigned a control coefficient, and according to the training data, high-imitation data, labels and control coefficients, a loss function is calculated and the control coefficient is adjusted, and the process of adjusting the control coefficient is continued until the output of the loss function meets the preset conditions; calculating the importance index of the operation operator, pruning the unimportant operation operator to obtain the pruned neural network model.

Description

[0001] This application is a divisional application. The application number of the original application is 202011006089.4, and the original application date is September 23, 2020. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to a neural network pruning method. Background Art

[0003] As one of the mainstream branches of machine learning, deep learning has been widely cited in image processing, face recognition and other fields. Convolutional Neural Networks (CNN) is one of the representative algorithms of deep learning. It has been widely used in large-scale computer vision applications (such as image classification, object detection and video analysis) and has achieved great success. Convolutional neural network models, especially deep neural network (DNN) models, have high requirements for computing power and memory. Edge devices such as mobile phones and wearable devices need to process deep learning-related tasks locally, and are limited by limited resources and power consumption as well as latency and cost considerations. Therefore, products based on deep learning are difficult to deploy on these edge devices.

[0004] In order to promote the application of deep learning-based products on edge devices, one approach is to design dedicated hardware accelerators for given computing tasks, and another approach is to simplify the neural network model to reduce its demand for computing power and memory, which is model compression. Model compression includes a variety of methods, such as pruning, quantization, low-rank factorization, and knowledge distillation. Among them, the network pruning method (Network Pruning) directly removes the entire convolution kernel, and can achieve acceleration without specific software and hardware design. It can reduce the number of network parameters and computational complexity without significantly affecting the network prediction accuracy, so it has received a lot of attention.

[0005] In the prior art, network pruning methods are mainly divided into two categories. The first method does not require training data, and determines the importance of different convolution kernels by pre-defining assumptions about the importance of convolution kernels. For example, it is assumed that convolution kernels with small norms are not important and removing some convolution kernels with small norms will not affect the performance of the network model too much. Another example is to calculate the geometric median of the convolution kernels in the network model and measure the importance of the convolution kernels according to the median, and delete the unimportant convolution kernels. The first method measures the importance of the convolution kernel based on a specific assumption, which cannot fully reflect the importance of the convolution kernel, and it is difficult to accurately mine the relationship between the convolution kernel and the network output, and the pruning process is interfered by a large number of irrelevant factors. The second method is a data-driven method, which requires the introduction of training data to learn the importance of different convolution kernels. For example, additional control coefficients are introduced to each convolution kernel, and the importance of different convolution kernels is measured by learning these control coefficients, and it is assumed that the convolution kernels corresponding to small control coefficients are not important. For example, the feature map of the middle layer of the network model is calculated by inputting an image, and the rank of the feature map is estimated based on the feature map, and the importance of the convolution kernel is measured based on the rank of the feature map. The second type of method may be very sensitive to slight changes in the input data in measuring the importance of the convolution kernel. In other words, a small fluctuation in the input data may have a greater impact on the pruning result, resulting in unstable results. It is also difficult to accurately mine the relationship between the convolution kernel and the network output, and the pruning process is interfered by a large number of irrelevant factors. Therefore, the technical problem in the prior art is how to minimize the interference of a large number of irrelevant factors on the pruning process, accurately mine the relationship between the convolution kernel and the network output, and improve the stability and reliability of the pruning results. Summary of the invention

[0006] The purpose of the present application is to provide a method for pruning a neural network model. The method comprises: obtaining labeled training data; obtaining high-imitation data based on the training data, wherein the high-imitation data has a similar distribution pattern to the training data, and the high-imitation data does not contain information related to the label; inputting the training data and the high-imitation data into the neural network model to obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layer of the neural network model, respectively, wherein the operation operators on each level of operation layer of the neural network model are each assigned a control coefficient, and the training features corresponding to a specific operation layer are calculated based on the high-imitation features and training features corresponding to the previous layer of the specific operation layer and the control coefficient corresponding to the operation operator of the specific operation layer. The invention relates to a method for determining a number of neural network models; calculating a loss function according to the training data, the high-imitation data, the label and the control coefficient; adjusting the control coefficient based on the loss function; continuing the process of adjusting the control coefficient based on the adjusted control coefficient, the training data and the high-imitation data until the output of the loss function meets the preset condition; calculating the importance index of each operation operator on each level of the operation layer of the neural network model based on the control coefficient obtained when the output of the loss function meets the preset condition; and pruning unimportant operation operators to obtain the pruned neural network model according to the calculated importance index of each operation operator on each level of the operation layer of the neural network model. In this way, by using the high-imitation features related to the high-imitation data as the control and supplement of the training features related to the training data, and by adjusting the control coefficient to affect the degree of participation of the high-imitation features in the subsequent calculation of the training features, the relationship between the network output and the operation operator is fully excavated, which is conducive to exploring the redundant operation operators in the network model, while minimizing the interference of irrelevant factors, and facilitating the stability and reliability of the pruning results.

[0007] In the first aspect, an embodiment of the present application provides a method for pruning a neural network model. The method comprises: obtaining labeled training data; obtaining high-imitation data based on the training data, wherein the high-imitation data has a similar distribution pattern to the training data, and the high-imitation data does not contain information related to the label; inputting the training data and the high-imitation data into the neural network model to respectively obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layers of the neural network model, wherein the operation operators on each level of operation layers of the neural network model are each assigned a control coefficient, and the training features corresponding to a specific operation layer are based on the high-imitation features and training features corresponding to the previous layer of the specific operation layer and the control coefficient corresponding to the operation operator of the specific operation layer. The invention relates to a method for determining a number; calculating a loss function according to the training data, the high-imitation data, the label and the control coefficient; adjusting the control coefficient based on the loss function; continuing the process of adjusting the control coefficient based on the adjusted control coefficient, the training data and the high-imitation data until the output of the loss function meets the preset condition; calculating the importance index of each operation operator on each operation layer of the neural network model based on the control coefficient obtained when the output of the loss function meets the preset condition; and pruning unimportant operation operators to obtain a pruned neural network model according to the calculated importance index of each operation operator on each operation layer of the neural network model.

[0008] The technical solution described in the first aspect uses the high-imitation features related to the high-imitation data as a control and supplement to the training features related to the training data, and adjusts the control coefficient to affect the degree of participation of the high-imitation features in the subsequent calculation of the training features, thereby fully exploring the relationship between the network output and the operation operator, which is beneficial to exploring the redundant operation operators in the network model, while minimizing the interference of irrelevant factors, which is beneficial to the stability and reliability of the pruning results.

[0009] According to the first aspect, in a possible implementation manner, the high imitation feature corresponding to the specific operation layer is obtained by performing an operation operation on the high imitation feature corresponding to a previous layer of the specific operation layer and an operation operator of the specific operation layer.

[0010] In this way, the high-imitation feature retains the attribute of not containing information related to the label, which is helpful to reduce the interference of irrelevant factors.

[0011] According to the first aspect, in a possible implementation method, the control coefficient includes a first control coefficient and a second control coefficient, and the training features corresponding to the specific operation layer are determined according to the high imitation features and training features corresponding to the previous layer of the specific operation layer and the control coefficient corresponding to the operation operator of the specific operation layer, including: the training features corresponding to the specific operation layer are obtained by adjusting the training features and the high imitation features corresponding to the previous layer of the specific operation layer according to the first control coefficient and the second control coefficient respectively, adding them together, and then performing an operation operation with the operation operator of the specific operation layer.

[0012] In this way, it is possible to influence the degree of participation of high-imitation features in subsequent calculation of training features by adjusting the control coefficient, and to reflect different degrees of participation through the first control coefficient and the second control coefficient, which is conducive to exploring redundant operation operators in the network model.

[0013] According to the first aspect, in a possible implementation manner, the first control coefficient and the second control coefficient corresponding to the same operation operator satisfy a constraint condition that the sum of the added coefficients is equal to 1.

[0014] In this way, the constraints of the control coefficients ensure that the changes of the first control coefficient and the second control coefficient are related, which is conducive to exploring redundant convolution kernels in the network model.

[0015] According to the first aspect, in a possible implementation method, the loss function is calculated according to the training data, the high-imitation data, the label and the control coefficient, including: calculating the loss function according to the training data, the high-imitation data, the label, a first control coefficient set, and a second control coefficient set, wherein the first control coefficient set includes first control coefficients corresponding to each operation operator on each level of the operation layer of the neural network model, and the second control coefficient set includes second control coefficients corresponding to each operation operator on each level of the operation layer of the neural network model.

[0016] In this way, by calculating the loss function based on all the control coefficients in the network and optimizing the loss function through label supervision, it is helpful to fully explore the relationship between the network output and the convolution kernel.

[0017] According to the first aspect, in a possible implementation method, the control coefficient is adjusted based on the loss function, including: based on the loss function, respectively adjusting the first control coefficient and the second control coefficient of each operation operator on each level of the neural network model.

[0018] In this way, the control coefficient is adjusted through the feedback of the loss function, which is conducive to fully exploring the relationship between the network output and the operation operator.

[0019] According to the first aspect, in a possible implementation method, based on the control coefficient obtained when the output of the loss function meets the preset conditions, the importance index of each operation operator on each operation layer of the neural network model is calculated, including: for each operation operator of the neural network model, the difference between the first control coefficient corresponding to the operation operator and the second control coefficient corresponding to the operation operator is calculated as the importance index corresponding to the operation operator.

[0020] In this way, by comparing the first control coefficient with the second control coefficient, the importance of the operation operator is effectively measured.

[0021] According to the first aspect, in a possible implementation method, based on the calculated importance index of the operation operators at each level of the operation layer of the neural network model, unimportant operation operators are pruned to obtain a pruned neural network model, including: determining an importance threshold according to a predetermined compression rate for the neural network model; comparing the importance index of each operation operator of the neural network model with the importance threshold, and judging the operation operator as unimportant when the importance index is less than the importance threshold.

[0022] In this way, the pruning process is controlled by combining a predetermined compression rate, thereby facilitating flexible configuration according to specific needs.

[0023] According to the first aspect, in a possible implementation, each level of the operation layer includes at least one convolution layer, the operation operator of the at least one convolution layer includes at least one convolution kernel, and the operation operation of the at least one convolution kernel is a convolution operation.

[0024] In this way, the application to the case including at least one convolution kernel and a convolution operation is achieved.

[0025] According to the first aspect, in a possible implementation, each stage of the operation layer includes at least one fully connected layer, the operation operator of the at least one fully connected layer includes at least one fully connected operator, and the operation operation of the at least one fully connected operator is a fully connected operation.

[0026] In this way, the application to the case including at least one full connection operator and a full connection operation is achieved.

[0027] According to the first aspect, in a possible implementation, the neural network model is a convolutional neural network model, the operation layers at each level are convolutional layers, the operation operators of the convolutional neural network are convolution kernels, and the operation operations of the operation operators of the convolutional neural network are convolution operations.

[0028] In this way, a convolutional neural network model is implemented in which all operation layers are convolutional layers.

[0029] According to the first aspect, in a possible implementation, the neural network model is a fully connected neural network model, the operation layers at each level are all fully connected layers, the operation operators of the fully connected neural network are all fully connected operators, and the calculation operations of the operation operators of the fully connected neural network are fully connected operations.

[0030] In this way, a fully connected neural network model is implemented in which all operation layers are fully connected layers.

[0031] According to the first aspect, in a possible implementation, the training data is a training image, the high imitation data is a high imitation image obtained by inputting the training image into a generative adversarial network model, and the neural network model is used for image processing, wherein the training data and the high imitation data are input into the neural network model to respectively obtain a plurality of training features and a plurality of high imitation features corresponding one-to-one to each level of operation layers of the neural network model, including: inputting the training image and the high imitation image into the neural network model to respectively obtain a plurality of training image features and a plurality of high imitation image features corresponding one-to-one to each level of operation layers of the neural network model.

[0032] In this way, the pruning process of the neural network model applied to image processing is realized.

[0033] According to the first aspect, in a possible implementation, an image processing system arranged on an edge device is characterized in that a neural network model used by the image processing system for image processing is obtained by pruning an uncompressed network model according to the described method.

[0034] In this way, by pruning the neural network model for image processing, it is beneficial to the productization and application of image processing systems on edge devices.

[0035] According to the first aspect, in a possible implementation method, the training data is a training speech signal, and the high-imitation data is a high-imitation speech signal obtained by inputting the training speech signal into a generative adversarial network model, wherein the neural network model is used for speech processing, wherein the training data and the high-imitation data are input into the neural network model to respectively obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layer of the neural network model, including: inputting the training speech signal and the high-imitation speech signal into the neural network model to respectively obtain a plurality of training speech signal features and a plurality of high-imitation speech signal features corresponding to each level of operation layer of the neural network model.

[0036] In this way, the pruning process of the neural network model applied to speech signal processing is realized.

[0037] According to the first aspect, in a possible implementation, a speech processing system is arranged on an edge device, wherein a neural network model used by the speech processing system for performing speech signal processing is obtained by pruning an uncompressed network model according to the described method.

[0038] In this way, by pruning the neural network model for speech signal processing, it is beneficial to the productization and application of speech processing systems on edge devices.

[0039] According to the first aspect, in a possible implementation, the training data is a training natural language signal, and the high-imitation data is a high-imitation natural language signal obtained by inputting the training natural language signal into a generative adversarial network model, wherein the neural network model is used for natural language processing, wherein the training data and the high-imitation data are input into the neural network model to respectively obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layer of the neural network model, including: inputting the training natural language signal and the high-imitation natural language signal into the neural network model to respectively obtain a plurality of training natural language signal features and a plurality of high-imitation natural language signal features corresponding to each level of operation layer of the neural network model.

[0040] In this way, the pruning process of the neural network model applied to natural language signal processing is realized.

[0041] According to the first aspect, in a possible implementation, a natural language processing system is arranged on an edge device, and a neural network model used by the natural language processing system for natural language signal processing is obtained by pruning an uncompressed network model according to the described method.

[0042] In this way, by pruning the neural network model of natural language signal processing, it is beneficial to the productization and application of natural language processing systems on edge devices.

[0043] According to the first aspect, in a possible implementation method, the training data is a training recommendation content signal, and the high imitation data is a high imitation recommendation content signal obtained by inputting the training recommendation content signal into a generative adversarial network model, wherein the neural network model is used to screen recommended content, wherein the training data and the high imitation data are input into the neural network model to respectively obtain a plurality of training features and a plurality of high imitation features corresponding to each level of operation layer of the neural network model, including: inputting the training recommendation content signal and the high imitation recommended content signal into the neural network model to respectively obtain a plurality of training recommendation content signal features and a plurality of high imitation recommended content signal features corresponding to each level of operation layer of the neural network model.

[0044] In this way, the pruning process of the neural network model applied to the recommended content signal processing is realized.

[0045] According to the first aspect, in a possible implementation, a content recommendation system is arranged on an edge device, wherein a neural network model used by the content recommendation system to screen recommended content is obtained by pruning an uncompressed network model according to the described method.

[0046] In this way, by pruning the neural network model that processes recommended content, it is beneficial to the productization and application of content recommendation systems on edge devices.

[0047] In the second aspect, an embodiment of the present application provides a compression method for a neural network model. The method includes: obtaining initial model parameters of an initial neural network model, wherein the initial model parameters include initial convolution kernels of each convolution layer of the initial neural network model; obtaining an experimental group image; obtaining a control group image based on the experimental group image and a trained adversarial generative model, wherein the control group image has a distribution similar to that of the experimental group image, and the control group image contains less target-related information than the experimental group image; inputting the control group image into the initial neural network model to obtain a first image feature corresponding to each convolution layer; assigning corresponding first control coefficients and second control coefficients to the initial convolution kernels of each convolution layer; inputting the experimental group image into the initial neural network model, and adjusting the input of the initial convolution kernels of each convolution layer according to the first control coefficient, the second control coefficient, and the first image feature to obtain a second image feature corresponding to each convolution layer; according to the experimental group image, the control group image, the The target related information included in the experimental group images, the first control coefficient set, the second control coefficient set, and the loss function are calculated, wherein the first control coefficient set includes the first control coefficients corresponding to the initial convolution kernels of the convolution layers at each level, and the second control coefficient set includes the second control coefficients corresponding to the initial convolution kernels of the convolution layers at each level; based on the loss function, the first control coefficient set and the second control coefficient set are adjusted; based on the adjusted first control coefficient set, the adjusted second control coefficient set, the experimental group images and the control group images, the adjustment process continues until the output of the loss function meets the preset conditions; based on the first control coefficient set and the second control coefficient set obtained when the output of the loss function meets the preset conditions, a part of the initial convolution kernels of the convolution layers at each level are selected as redundant convolution kernels; the selected redundant convolution kernels are cut off from the initial neural network model to obtain a compressed neural network model.

[0048] The technical solution described in the second aspect uses the first image features related to the control group images as a control and supplement to the second image features related to the experimental group images, and adjusts the control coefficient to affect the degree of participation of the first image features in the subsequent calculation of the second image features, thereby fully exploring the relationship between the network output and the convolution kernel, which is beneficial to exploring redundant convolution kernels in the network model, while minimizing the interference of irrelevant factors, which is beneficial to the stability and reliability of the pruning results.

[0049] According to the second aspect, in a possible implementation method, the input of the initial convolution kernel of each level of convolution layer is adjusted according to the first control coefficient, the second control coefficient, and the first image feature to obtain the second image feature corresponding to each level of convolution layer one by one, including: multiplying the first image feature corresponding to the initial convolution kernel and the input feature received by the initial convolution kernel with the first control coefficient and the second control coefficient respectively, and then adding the results as the input of the adjusted initial convolution kernel.

[0050] In this way, it is possible to influence the degree of participation of the first image feature in the subsequent calculation of the second image feature by adjusting the control coefficient, and to reflect the different degrees of participation through the first control coefficient and the second control coefficient, which is conducive to exploring redundant convolution kernels in the network model.

[0051] According to the second aspect, in a possible implementation manner, the first control coefficient and the second control coefficient corresponding to the same convolution kernel satisfy the constraint that the sum of their addition is equal to 1.

[0052] In this way, the constraints of the control coefficients ensure that the changes of the first control coefficient and the second control coefficient are related, which is conducive to exploring redundant convolution kernels in the network model.

[0053] According to the second aspect, in a possible implementation method, based on the first control coefficient set and the second control coefficient set obtained when the output of the loss function meets the preset conditions, a part of the initial convolution kernels of the convolution layers at each level are selected as redundant convolution kernels, including: when the difference between the second control coefficient corresponding to the initial convolution kernel and the first control coefficient corresponding to the initial convolution kernel is less than a preset threshold, it is judged that the initial convolution kernel is a redundant convolution kernel.

[0054] In this way, by comparing the first control coefficient with the second control coefficient, the importance of the convolution kernel is effectively measured, and the pruning process is controlled by a preset threshold, which is conducive to flexible configuration according to specific needs.

[0055] According to the second aspect, in a possible implementation method, the first control coefficient set and the second control coefficient set obtained when the output of the loss function meets the preset conditions respectively indicate the irrelevance and importance of the initial convolution kernels of the convolution layers at each level for achieving the goal.

[0056] In this way, the interference of irrelevant factors is minimized.

[0057] According to the second aspect, in a possible implementation, the image processing system includes a processor and a memory, the memory stores computer instructions, the computer instructions are executed by the processor to perform image processing operations, and the image processing operations are based on the neural network model compressed by the described method.

[0058] In this way, by using the compressed neural network model as the basis for image processing operations, it is helpful to reduce latency and reduce the demand for computing power and memory.

[0059] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the processor performs the following operations: obtain training data with labels; obtain high-imitation data based on the training data, wherein the high-imitation data has a similar distribution pattern to the training data, and the high-imitation data does not contain information related to the label; input the training data and the high-imitation data into the neural network model to obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layers of the neural network model, respectively, wherein the operation operators on each level of operation layers of the neural network model are each assigned a control coefficient, and the training features corresponding to a specific operation layer are based on the training features corresponding to the previous layer of the specific operation layer. The high-imitation features and training features of the neural network model and the control coefficient corresponding to the operation operator of the specific operation layer are determined; the loss function is calculated according to the training data, the high-imitation data, the label and the control coefficient; the control coefficient is adjusted based on the loss function; based on the adjusted control coefficient, the training data and the high-imitation data, the control coefficient adjustment process is continued until the output of the loss function meets the preset conditions; based on the control coefficient obtained when the output of the loss function meets the preset conditions, the importance index of each operation operator on each operation layer of the neural network model is calculated; and according to the calculated importance index of each operation operator on each operation layer of the neural network model, unimportant operation operators are pruned to obtain a pruned neural network model.

[0060] The technical solution described in the third aspect uses the high-imitation features related to the high-imitation data as a control and supplement to the training features related to the training data, and adjusts the control coefficient to affect the degree of participation of the high-imitation features in the subsequent calculation of the training features, thereby fully exploring the relationship between the network output and the operation operator, which is beneficial to exploring the redundant operation operators in the network model, while minimizing the interference of irrelevant factors, which is beneficial to the stability and reliability of the pruning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0062] Figure 1 The application scenario of the neural network pruning method provided in the embodiment of the present application is shown.

[0063] Figure 2 A schematic diagram of the process of obtaining real data and high-simulation data provided in an embodiment of the present application is shown.

[0064] Figure 3 It shows that the embodiment of the present application provides Figure 2 Schematic diagram of the real data and high-simulation data being input into the neural network model for transmission.

[0065] Figure 4 A flowchart of a neural network pruning method according to an implementation method provided in an embodiment of the present application is shown.

[0066] Figure 5 A schematic flow chart of a method for compressing a network model for image processing provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0067] The embodiment of the present application provides a method for pruning a neural network model. The method includes: obtaining labeled training data; obtaining high-imitation data based on the training data, wherein the high-imitation data has a similar distribution pattern to the training data, and the high-imitation data does not contain information related to the label; inputting the training data and the high-imitation data into the neural network model to obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layer of the neural network model, respectively, wherein the operation operators on each level of operation layer of the neural network model are each assigned a control coefficient, and the training features corresponding to a specific operation layer are based on the high-imitation features and training features corresponding to the previous layer of the specific operation layer and the control coefficient corresponding to the operation operator of the specific operation layer. The invention relates to a method for determining a number of neural network models; calculating a loss function according to the training data, the high-imitation data, the label and the control coefficient; adjusting the control coefficient based on the loss function; continuing the process of adjusting the control coefficient based on the adjusted control coefficient, the training data and the high-imitation data until the output of the loss function meets the preset condition; calculating the importance index of each operation operator on each level of the operation layer of the neural network model based on the control coefficient obtained when the output of the loss function meets the preset condition; and pruning unimportant operation operators to obtain the pruned neural network model according to the calculated importance index of each operation operator on each level of the operation layer of the neural network model. In this way, by using the high-imitation features related to the high-imitation data as the control and supplement of the training features related to the training data, and by adjusting the control coefficient to affect the degree of participation of the high-imitation features in the subsequent calculation of the training features, the relationship between the network output and the operation operator is fully excavated, which is conducive to exploring the redundant operation operators in the network model, while minimizing the interference of irrelevant factors, and facilitating the stability and reliability of the pruning results.

[0068] The embodiments of the present application can be used in the following application scenarios, including but not limited to, a processing system based on a neural network model deployed on an edge device (such as a mobile phone, a wearable device, etc.), for large-scale computer vision applications (such as image classification, target detection, and video analysis, etc.), or for speech signal processing, natural language processing, recommendation systems, and other situations where the neural network model needs to be compressed to remove redundant operation operators (such as convolution kernels for convolution operations or fully connected operators for fully connected operations) due to limited resources and latency requirements.

[0069] The embodiments of the present application can be adjusted and improved according to the specific application environment and are not specifically limited here.

[0070] In order to enable those skilled in the art to better understand the present application, the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0071] See also Figure 1 , Figure 1 The following illustrates the application scenario of the neural network pruning method provided in the embodiment of the present application. Figure 1 As shown, in step S100, an initial neural network model is obtained. Specifically, the initial model parameters of the initial neural network model can be obtained. The initial model parameters include the initial convolution kernels of each level of convolutional layer of the initial neural network model, and may also include other common structures such as pooling layers and fully connected layers, and may also include the internal architecture of the initial neural network model. The initial neural network model can be an algorithm model based on deep learning, which can be used for large-scale computer vision applications, such as to implement functions such as image classification, target detection, and video analysis, and can also be used to implement functions such as screening recommended content, natural language processing, and speech signal analysis. In step S110, the initial neural network model obtained in step S100 is pruned, such as removing unimportant or redundant convolution kernels, or, for example, removing unimportant fully connected layers or some of the fully connected operations therein.

[0072] It should be understood that the initial neural network model can be understood to include one or more convolutional layers, one or more fully connected layers, and other possible structures. Among them, the convolutional layer, the fully connected layer and other similar structures can be called the operation layer of the initial neural network model, and the operation layer has corresponding operation operators to complete the corresponding operation. For example, there is a convolution kernel on the convolution layer to complete the convolution operation, or there is a fully connected operator on the fully connected layer to complete the fully connected operation, or other structures provide operation operations such as linear superposition. The following takes the operation layer as a convolution layer, the operation operator as a convolution kernel, and the operation operation as a convolution operation as an example. The following embodiment takes the convolution kernel of the convolution layer of the convolutional neural network model as an example, but it should be understood that it can be applied to neural network models with other operation modes and other architectures.

[0073] According to the network pruning process described in detail in the embodiment of the present application below, the pruning process of step S110 realizes accurate mining of the relationship between the convolution kernel and the network output, and minimizes the interference of irrelevant factors on the pruning process, thereby greatly simplifying the initial neural network model while retaining the network prediction accuracy, and improving the stability and reliability of the pruning results. In step S120, the pruned neural network model is deployed to the edge device. The edge device can be a mobile phone, a wearable device, or other device that needs to process deep learning-related tasks locally and is limited by local limited resources. For example, if the initial neural network model is a model for target detection, the initial neural network model can be pruned and deployed to devices such as security cameras for target detection, thereby utilizing the simplified neural network model to occupy less computing power and memory, which is also beneficial for saving energy and shortening latency.

[0074] In some exemplary embodiments, a step may be added between step S110 and step S120 to retrain the pruned neural network model, thereby fine-tuning the model parameters to further improve network performance, such as fine-tuning the weight coefficients of the convolution kernels of each convolution layer by retraining.

[0075] See also Figure 2 , Figure 2 The flowchart of obtaining real data and high-simulation data provided by the embodiment of the present application is shown. Figure 2 As shown, the process includes the following steps.

[0076] Step S200: Acquire real data.

[0077] Among them, real data refers to data with labels under supervised learning, which can be training images containing information about the target, or can be public training data sets, benchmark data sets and verification data sets. For example, real data can be the benchmark data set CIFAR-10 or the large-scale data set ImageNet (ILSVRC-2012). Real data can have a certain distribution law, such as conforming to the Gaussian distribution law. Real data can be a training data set configured according to a specific application scenario. For example, when a neural network model is used for image recognition, real data can be a general training image set for training a neural network model for image recognition. When a neural network model is used for speech recognition and processing, real data can be a general training data set for training artificial intelligence to process speech signals. Similarly, when a neural network model is used for natural language processing, real data can be a general training natural language signal for training a neural network model for processing natural language, such as the original text and translated text used in machine translation training. When a neural network model is used to screen recommended content for automatic push, real data can be training materials for training a neural network model for an automatic recommendation system, such as product materials used to automatically push to e-commerce users. These can be determined according to specific application scenarios and are not specifically limited here.

[0078] Step S202: Generate high-fidelity data through an adversarial generative model.

[0079] Among them, the adversarial generation model is a neural network model based on the Generative Adversarial Networks (GAN) algorithm. Taking the generation of pictures as an example, the principle of the GAN algorithm is to generate pictures that are as realistic as possible through the game between the generation network and the discriminant network. Therefore, given the real data obtained in step S200, high-imitation data can be generated by training the adversarial generation model.

[0080] Step S204: Determine whether the generated high-imitation data meets the requirements through the loss function.

[0081] Among them, the loss function needs to be designed for specific requirements, so as to optimize the adversarial generation model according to the loss function. There are two requirements for high-imitation data: the first is that the high-imitation data must have a distribution law similar to that of the real data. Here, similar distribution laws can be understood as the similar distribution of real data and high-imitation data in high-dimensional space, such as having the characteristics of Gaussian distribution. For example, assuming that the real data conforms to the Gaussian distribution law, the high-imitation data must also satisfy the Gaussian distribution law. The second is that the high-imitation data contains as little target-related information as possible, or information related to the labels carried in the real data. Ideally, the high-imitation data does not contain target-related information. Here, the target-related information corresponds to information related to the importance of the convolution kernel. The fact that the high-imitation data does not contain target-related information means that the control coefficient or importance measurement information obtained later based on the high-imitation data is also irrelevant to the importance of the convolution kernel. Therefore, by using real data as the experimental group to obtain information related to the importance of the convolution kernel, and using high-simulation data as the control group to obtain information unrelated to the importance of the convolution kernel, it is possible to reduce irrelevant factors by comparing the responses caused by real data and high-simulation data respectively. For example, the measure of the importance of the convolution kernel that contains target-related information can be subtracted from the measure of the importance of the convolution kernel that does not contain target-related information.

[0082] In step S204, the loss function of the GAN algorithm can be reasonably designed to measure whether the generated high-imitation data meets the above two requirements. For example, the degree of target-related information included in the high-imitation data can be measured by numerical means. When the measurement value is less than a certain threshold, it can be determined that the high-imitation data meets the above two requirements. The relationship between the real data and the high-imitation data can be expressed as formula (1).

[0083]

[0084] Among them, X represents the real data, represents the high-imitation data, and the function in formula (1) represents the mapping from the real data to the high-imitation data. According to the judgment result in step S204, if the loss function determines that the high-imitation data does not meet the above requirements, step S208 is executed; if the loss function determines that the high-imitation data meets the above requirements, step S206 is executed.

[0085] Step S206: Output the real data and the high-simulation data.

[0086] The high-imitation data is determined to meet the above requirements in step S204, that is, it has a distribution pattern similar to the real data and does not contain target-related information or contains target-related information to a degree less than a preset threshold.

[0087] Step S208: Optimize the adversarial generation model according to feedback of the loss function.

[0088] In step S204, it is determined that the high-imitation data does not meet the above requirements, so the loss function provides feedback to optimize the adversarial generation model. After the adversarial generation model is optimized, step S202 is executed again, and new high-imitation data is generated using the optimized adversarial generation model, and then it is determined again in step S204 whether the new high-imitation data meets the requirements. In this way, through the loop of steps S202, S204 and S208, a loop of generating high-imitation data, judging whether it meets the requirements, and optimizing the model according to feedback is formed. After multiple iterations, high-imitation data that meets the requirements is finally generated and output in step S206.

[0089] In some exemplary embodiments, instead of judging whether the requirements are met through a loss function, the latest high-simulation data can be output after a preset number of iterations are reached and the number of high-simulation data and the optimization model are cyclically generated and the number of iterations is reached.

[0090] See also Figure 3 , Figure 3 It shows that the embodiment of the present application provides Figure 2 The schematic diagram of the real data and high-simulation data being input into the neural network model for transmission is shown in FIG. It should be understood that Figure 3 The neural network model can be understood as including one or more convolutional layers, one or more fully connected layers, and other possible structures. Among them, the convolutional layer, the fully connected layer and other similar structures can be called the operation layer of the neural network model, and the operation layer has corresponding operation operators to complete the corresponding calculation operations. For example, there is a convolution kernel on the convolution layer to complete the convolution operation, or there is a fully connected operator on the fully connected layer to complete the fully connected operation, or other structures provide calculation operations such as linear superposition. The following example takes the operation layer as a convolution layer, the operation operator as a convolution kernel, and the calculation operation as a convolution operation. The following embodiment takes the convolution kernel of the convolution layer of the convolutional neural network model as an example, but it should be understood that it can be applied to neural network models with other operation modes and other architectures. As Figure 3 As shown in the figure, the neural network model includes convolutional layers at various levels, each of which contains one or more convolutional kernels, and each convolutional kernel has its own weight coefficient. Figure 3In the figure, for the purpose of simplicity, only the first convolution layer 300, the second convolution layer 302, the Lth convolution layer 304, and the L+1th convolution layer 306 are shown at each level of the convolution layer. It should be understood that the neural network model may also include other convolution layers not shown, and may also include common structures such as pooling layers or fully connected layers, or may include more complex structures such as Inception modules, residual blocks, etc. These may be determined according to specific application scenarios and are not specifically limited here. In addition, the first convolution layer, the second convolution layer, and the Lth convolution layer are all from the perspective of data transmission, wherein the first convolution layer refers to the convolution layer in which the neural network model performs the first convolution operation or feature extraction on the input information according to its hierarchical structure, and the second convolution layer is the convolution layer that performs the second convolution operation or feature extraction after the first convolution layer, and so on, the Lth convolution layer is the convolution layer that performs the convolution operation or feature extraction for the Lth time. It should be understood that an input layer may be configured before the first convolution layer to perform input feature standardization operations, and a pooling layer and a fully connected layer may be configured after the first convolution layer. More complex structures or modules may be configured in the first convolution layer and the second convolution layer. These can be determined according to specific application scenarios and are not specifically limited here.

[0091] like Figure 3 As shown in Figure 1, the high-fidelity data and real data are input into the same neural network model, but have different transmission processes. Figure 3 The first convolutional layer 300 appears twice in , which should be understood as the same first convolutional layer 300, but the convolution operation is performed on the real data and the high-simulation data respectively. Figure 3 The second convolution layer 302 and the Lth convolution layer 304 appear twice in the figure. These are convolution operations for the transmission process of real data and high-imitation data respectively. The details of the transmission process of high-imitation data are as follows: the high-imitation data is input to the first convolution layer 300 for convolution operation, the output result of the first convolution layer 300 is forwarded to the second convolution layer 302 for convolution operation, and the output result of the second convolution layer 302 is sequentially transmitted to the Lth convolution layer 304 for convolution operation. Among them, the Lth convolution layer 304 has three convolution kernels, namely W1, L, W2, L, W3, L. The three convolution kernels of the Lth convolution layer 304 perform convolution operations on the input feature data respectively and then output them.

[0092] Please continue reading Figure 3, the details of the real data transmission process are as follows: the real data is input to the first convolution layer 300 for convolution operation, and the convolution operation result of the real data transmission process output by the first convolution layer 300 is input to the modulation layer 310 output by the first convolution layer. The modulation layer 310 output by the first convolution layer receives the convolution operation result of the real data transmission process output by the first convolution layer 300, and also receives the convolution operation result of the high-imitation data transmission process output by the first convolution layer 300, performs mixed modulation on the two, and then transmits the mixed modulation result to the second convolution layer 302 for convolution operation. The modulation layer 310 output by the first convolution layer is controlled by the modulation layer control parameter distributor 340, and the modulation layer control parameter distributor 340 sets the control parameters for mixed modulation to affect the result of mixed modulation. Similarly, the convolution operation result of the real data transmission process output by the second convolution layer 302 is transmitted to the modulation layer 320 output by the second convolution layer. The modulation layer 320 output by the second convolution layer receives the convolution operation result of the real data transmission process output by the second convolution layer 302, and also receives the convolution operation result of the high-imitation data transmission process output by the second convolution layer 302, performs mixed modulation on the two, and then transmits the mixed modulation result to the next convolution layer for convolution operation. The modulation layer 320 output by the second convolution layer is controlled by the modulation layer control parameter distributor 340, and the modulation layer control parameter distributor 340 sets the control parameters for mixed modulation to affect the result of mixed modulation. Therefore, the transmission process of the real data after being input into the neural network model is different from the transmission process of the high-imitation data in that the output of each convolution layer must be passed on after mixed modulation of a modulation layer. The result of mixed modulation is obtained based on the convolution operation result of the real data transmission process and the convolution operation result of the high-imitation data transmission process output by the same convolution layer, and then according to the control coefficient for mixed modulation allocated by the modulation layer control parameter distributor 340.

[0093] The following takes the modulation layer 330 output by the Lth convolution layer between the Lth convolution layer and the L+1th convolution layer as an example to illustrate the structure of the hybrid modulation. The Lth convolution layer 304 has three convolution kernels, namely W1, L, W2, L, W3, and L. The three convolution kernels of the Lth convolution layer 304 respectively perform convolution operations on the input feature data and then output them. Correspondingly, the modulation layer 330 output by the Lth convolution layer also includes three modulators, convolution kernel W1, L output modulator 332, convolution kernel W2, L output modulator 334, and convolution kernel W3, L output modulator 336, which correspond to the three convolution kernels W1, L, W2, L, W3, and L of the Lth convolution layer 304, respectively. Among them, the convolution kernel W1, L performs a convolution operation on the input features received about the real data transmission process, outputs the convolution operation result to the corresponding convolution kernel W1, L output modulator 332, and then mixes and modulates with the result of the convolution operation performed by the convolution kernel W1, L on the input features received about the high-simulation data transmission, and then passes it to the L+1 convolution layer. Similarly, the convolution kernel W2, L performs a convolution operation on the input features received about the real data transmission process, outputs the convolution operation result to the corresponding convolution kernel W2, L output modulator 334, and then mixes and modulates with the result of the convolution operation performed by the convolution kernel W2, L on the input features received about the high-simulation data transmission, and then passes it to the L+1 convolution layer. The convolution kernel W3,L performs a convolution operation on the received input features of the real data transmission process, and outputs the convolution operation result to the corresponding convolution kernel W3,L output modulator 336, which is then mixed and modulated together with the result of the convolution operation performed by the convolution kernel W3,L on the received input features of the high-simulation data transmission, and then passed to the L+1th convolution layer.

[0094] Please continue reading Figure 3 In some exemplary embodiments, the convolution operation results of each convolution kernel outputted in the high-simulation data transfer process may be multiple high-simulation features corresponding to each convolution kernel one by one, and the convolution operation results of each convolution kernel outputted in the real data transfer process may be multiple real features corresponding to each convolution kernel one by one. Figure 3 As shown in the figure, the process of transmitting real data and high-imitation data layer by layer in the neural network model is different. The high-imitation features corresponding to a specific convolutional layer are obtained by convolving the high-imitation features corresponding to the previous layer of the specific convolutional layer with the convolution kernel of the specific convolutional layer. For example, Figure 3The high-imitation features output by the second convolution layer 302 are obtained by convolving the high-imitation features output by the first convolution layer 300 with the convolution kernel of the second convolution layer 302. The real features corresponding to the specific convolution layer are obtained by outputting the real features and high-imitation features corresponding to the previous layer of the specific convolution layer to the modulation layer corresponding to the previous layer of the specific convolution layer for mixed modulation, and then convolving with the convolution kernel of the specific convolution layer. For example, Figure 3 The real features output by the second convolution layer 302 are obtained by outputting the real features and the high-imitation features output by the first convolution layer 300 to the modulation layer 310 of the first convolution layer output for mixed modulation and then convolving with the convolution kernel of the second convolution layer 302. Figure 3 In some exemplary embodiments, the real data is the experimental group image, the high-imitation data is the control group image, and the control group image contains less target-related information than the experimental group image. The first image features corresponding to each level of convolution layer are obtained by inputting the control group image into the neural network model. The second image features are obtained by inputting the experimental group image into the neural network model and adjusting the input of the convolution kernel of each level of convolution layer according to the first image features and the corresponding control coefficients. Here, the adjustment of the control coefficient is realized by the modulator in the modulation layer before the input of each convolution kernel.

[0095] Please continue reading Figure 3 , Figure 3 The Lth convolution layer 304 and the modulation layer 330 of the corresponding Lth convolution layer output respectively have three convolution kernels and three modulators. It should be understood that Figure 3 The number of convolution kernels of each convolution layer of the neural network shown is for illustrative purposes only and should not be construed as limiting. In some exemplary embodiments, a specific convolution layer of the neural network model may have one or more convolution kernels, each of which has a corresponding modulator. The hybrid modulation performed at each modulation layer can be achieved by setting a first control coefficient and a second control coefficient. For example, the real feature corresponding to a specific convolution layer is obtained by adding the real feature and the high imitation feature corresponding to the previous layer of the specific convolution layer after adjusting them according to the first control coefficient and the second control coefficient, respectively, and then convolving them with the convolution kernel of the specific convolution layer. Among them, the modulation layer control parameter distributor 340 allocates the corresponding first control coefficient and the second control coefficient to each modulator of each modulation layer, thereby realizing various hybrid modulation operations regarding the transmission process of real data.

[0096] See also Figure 4 , Figure 4 A flowchart of a neural network pruning method according to an implementation method provided in an embodiment of the present application is shown. Figure 4 The method shown includes the following steps.

[0097] Step S400: Obtain training data with labels and obtain high-quality imitation data based on the training data.

[0098] Among them, training data refers to data with labels under supervised learning, which can be training images containing information about the target, or can be public training data sets, benchmark data sets and verification data sets. For example, the training data can be the benchmark data set CIFAR-10 or the large-scale data set ImageNet (ILSVRC-2012). The training data has a certain distribution law, such as conforming to the Gaussian distribution law. The training data can be a training data set configured according to a specific application scenario. For example, when the neural network model is used for image recognition, the training data can be a general training image set for training the neural network model for image recognition, and when the neural network model is used for speech recognition and processing, the training data can be a general training data set for training artificial intelligence to process speech signals. Similarly, when the neural network model is used for natural language processing, the real data can be a general training natural language signal for training the neural network model for processing natural language, such as the original text and translated text used in the training of machine translation. When the neural network model is used to screen recommended content for automatic push, the real data can be the training data of the neural network model for training the automatic recommendation system, such as the product data used for automatic push to e-commerce users. These can be determined according to the specific application scenario and are not specifically limited here.

[0099] There are two requirements for high-imitation data: the first is that the high-imitation data must have a distribution law similar to that of the training data. Here, similar distribution laws can be understood as the similar distribution of training data and high-imitation data in high-dimensional space, such as having the characteristics of Gaussian distribution. For example, assuming that the training data conforms to the Gaussian distribution law, the high-imitation data must also satisfy the Gaussian distribution law. The second is that the high-imitation data contains as little target-related information as possible, or information related to the labels in the training data. Ideally, the high-imitation data does not contain target-related information. Here, the target-related information corresponds to information related to the importance of the convolution kernel. The fact that the high-imitation data does not contain target-related information means that the control coefficient or importance measurement information obtained later based on the high-imitation data is also irrelevant to the importance of the convolution kernel.

[0100] See also Figure 2 and Figure 4 , Figure 4 The training data can correspond to Figure 2 The real data Figure 4 The high-fidelity data corresponding to Figure 2 Therefore, the relationship between training data and high-simulation data can be expressed as the above formula (1), where X represents training data, X ～represents high-simulation data, and the function in formula (1) can be used to represent the mapping from training data to high-simulation data.

[0101] Step S402: input the training data and the high-imitation data into the neural network model to obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layers of the neural network model, respectively, wherein the operation operators on each level of operation layers of the neural network model are respectively assigned a control coefficient, and the training features corresponding to a specific operation layer are determined according to the high-imitation features and training features corresponding to the previous layer of the specific operation layer and the control coefficient corresponding to the operation operator of the specific operation layer.

[0102] Among them, the high imitation feature and the training feature correspond to the intermediate output result generated during the layer-by-layer transmission of the high imitation data in the neural network model and the intermediate output result generated during the layer-by-layer transmission of the training data in the neural network model. The high imitation feature corresponding to the specific operation layer is obtained by performing an operation operation on the high imitation feature corresponding to the previous layer of the specific operation layer and the operation operator of the specific operation layer. The control coefficient includes a first control coefficient and a second control coefficient. The training feature corresponding to the specific operation layer is obtained by adding the training feature and the high imitation feature corresponding to the previous layer of the specific operation layer according to the first control coefficient and the second control coefficient respectively, and then performing an operation operation on the operation operator of the specific operation layer. The following example takes the operation layer as a convolution layer, the operation operator as a convolution kernel, and the operation operation as a convolution operation. The following embodiment takes the convolution kernel of the convolution layer of the convolution neural network model as an example, but it should be understood that it can be applied to neural network models of other operation modes and other architectures. For example, in addition to the convolution layer and the fully connected layer, the neural network model has other similar structures that can be called the operation layer of the neural network model, and there are corresponding operation operators on the operation layer to complete the corresponding operation operations. For example, there are convolution kernels on the convolution layer to complete the convolution operation, or there are fully connected operators on the fully connected layer to complete the fully connected operation, or other structures provide operations such as linear superposition.

[0103] Please continue reading Figure 4 , the high-imitation features corresponding to the L+1th convolutional layer are obtained by convolving the high-imitation features corresponding to the Lth layer with the convolution kernel of the L+1th convolutional layer, and the real features corresponding to the L+1th convolutional layer are obtained by adding the training features and high-imitation features corresponding to the Lth layer after adjusting them according to the first control coefficient and the second control coefficient, and then convolving them with the convolution kernel of the L+1th convolutional layer. The high-imitation features and training features corresponding to the Lth convolutional layer can be expressed as formulas (2) and (3).

[0104]

[0105] Where X represents the training data, represents high-simulation data, A^l represents the training features corresponding to the Lth convolutional layer, and A ～ ^l represents the high-simulation feature corresponding to the Lth convolutional layer, W^(1:l) represents the convolution kernel of the Lth convolutional layer, and the functions of formulas (2) and (3) represent the convolution operation. The high-simulation feature A corresponding to the L+1th convolutional layer ～ ^(l+1) is the high-level feature A corresponding to the Lth convolutional layer ～ The training features corresponding to the L+1 convolution layer satisfy formula (4).

[0106]

[0107] Among them, A^l represents the training features corresponding to the Lth convolutional layer, A^(l+1) represents the training features corresponding to the L+1th convolutional layer, and A ～ ^l represents the high-imitation feature corresponding to the Lth convolutional layer, β^l represents the first control coefficient of the convolution kernel of the Lth convolutional layer, β ～ ^l represents the second control coefficient of the convolution kernel of the Lth convolution layer, W^(1:l+1) represents the convolution kernel of the L+1th convolution layer, ⊙ represents the element-wise multiplication operation, and * represents the convolution operation. As shown in formula (4), the training features and high-imitation features corresponding to the Lth convolution layer are respectively multiplied element-wise with the first control coefficient and the second control coefficient, that is, they are adjusted according to the first control coefficient and the second control coefficient and then added. The sum of the two is then convolved with the convolution kernel of the L+1th convolution layer to obtain the training features corresponding to the L+1th convolution layer. It should be understood that the convolution kernels on each level of the convolutional neural network model are each assigned a control coefficient, so the first control coefficient and the second control coefficient are both for a specific convolution kernel. And for the same convolution kernel, the first control coefficient and the second control coefficient satisfy the constraints represented by formula (5).

[0108]

[0109] Among them, β^l represents the first control coefficient of the convolution kernel of the Lth convolution layer, and β~^l represents the second control coefficient of the convolution kernel of the Lth convolution layer. In other words, the first control coefficient and the second control coefficient corresponding to the same convolution kernel satisfy the constraint that the sum is equal to 1. Assuming that there are N convolution kernels, each convolution kernel has a corresponding pair of first control coefficients and second control coefficients, and each pair of first control coefficients and second control coefficients satisfies the constraint shown in formula (5).

[0110] Step S404: Calculate the loss function based on the training data, high-resolution data, labels, and control coefficients.

[0111] Among them, the labeled training data obtained in step S400 and the high-imitation data obtained based on the training data, the labels can be understood as target-related information. Because the high-imitation data does not contain target-related information, and the target-related information corresponds to information related to the importance of the convolution kernel, the high-imitation data has the property that it does not contain information related to the importance of the convolution kernel. It should be understood that the multiple training features and multiple high-imitation features obtained by inputting the training data and the high-imitation data into the convolutional neural network model in step S402 correspond to the convolution kernels of each convolution layer, respectively, and each convolution kernel is also assigned a corresponding control coefficient including a first control coefficient and a second control coefficient. In other words, the high-imitation features and the training features correspond to the intermediate output results generated during the layer-by-layer transmission of the high-imitation data in the convolutional neural network model and the intermediate output results generated during the layer-by-layer transmission of the training data in the convolutional neural network model. Combined with the method of generating high-imitation features mentioned in step S402, that is, the high-imitation features corresponding to the specific convolutional layer are obtained by convolving the high-imitation features corresponding to the previous layer of the specific convolutional layer with the convolution kernel of the specific convolutional layer, which means that the high-imitation features only involve the convolution operation of the convolution kernels of each convolutional layer in the convolutional neural network model. Therefore, the property of the high-imitation data not containing information related to the importance of the convolution kernel can be maintained during the network transmission process. The high-imitation data and the high-imitation features generated by the layer-by-layer transmission of the high-imitation data in the convolutional neural network model both maintain the property of not containing information related to the importance of the convolution kernel. For example, Figure 3 The high-simulation data transmission process shown is from the first convolution layer 300 to the second convolution layer 302 to the Lth convolution layer 304. In this way, the high-simulation features generated during the network transmission process are consistent with the high-simulation data input to the model in terms of the degree of target-related information contained therein, that is, if the high-simulation data does not contain information related to the importance of the convolution kernel, the high-simulation features do not contain information related to the importance of the convolution kernel. In this way, by generating high-simulation data and high-simulation features as a comparison and supplement to the training data and training features, the relationship between the network output and the convolution kernel can be fully explored, and the redundant convolution kernels in the network model can be explored.

[0112] Compared with high-imitation data and high-imitation features, the training data with labels also contains information related to the importance of the convolution kernel. Combined with the generation method of the training features mentioned in step S402, that is, the training features corresponding to the specific convolution layer are obtained by adjusting the training features and high-imitation features corresponding to the previous layer of the specific convolution layer according to the first control coefficient and the second control coefficient respectively, and then adding them to the convolution kernel of the specific convolution layer. This means that the generation of training features not only involves the training features output by the previous layer but also involves the high-imitation features output by the previous layer, and the participation degree of the high-imitation features is adjusted by the second control coefficient. Combined with the constraint conditions between the first control coefficient and the second control coefficient mentioned in formula (5), this means that the high-imitation features and training features corresponding to the same convolution kernel, the participation degree of each in the calculation of the subsequent feature map is adjusted by the paired first control coefficient and the second control coefficient, and it is manifested as "one increases while the other decreases". That is to say, when the participation degree of the high-imitation feature increases, it means that the second control coefficient increases, which means that the corresponding first control coefficient must be constructed due to the influence of the constraint conditions, thereby causing the participation degree of the corresponding training feature to decrease, and vice versa. In this way, a loss function can be designed to optimize the control coefficients, thereby mining redundant convolution kernels. The loss function can be expressed as formula (6).

[0113]

[0114] Among them, function E is a loss function, X represents training data, X~ represents high-imitation data, Y represents label, β and β~ represent all control coefficients in the network, β^l represents the first control coefficient of the convolution kernel of the Lth convolution layer, β~^l represents the second control coefficient of the convolution kernel of the Lth convolution layer, assuming that there are N convolution kernels and the paired first control coefficient and second control coefficient of each convolution kernel satisfy the constraint that the sum is equal to 1. The input of the loss function E is the output of the convolution neural network model. As mentioned above, the output of the convolution neural network model is affected by the training data, high-imitation data and all control coefficients in the network. Because the training data and high-imitation data are unchanged, the loss function E can measure the influence of different control coefficients on the output of the convolution neural network model according to the label Y, that is, it realizes the optimization of the control coefficient according to the supervision of the label Y. Formula (6) reflects the calculation of the loss function according to the training data, the high-imitation data, the label and the control coefficient. In this way, by designing the loss function, the control coefficients of each convolution kernel can be optimized under the supervision of the label Y, including optimizing the first control coefficient and the second control coefficient under the premise of satisfying the constraint. For example, the loss function can be allowed to converge by adjusting the control coefficient according to the label Y.

[0115] In some exemplary embodiments, the loss function may be calculated based on the training data, the high-imitation data, the label, a first control coefficient set, and a second control coefficient set. The first control coefficient set includes first control coefficients corresponding to convolution kernels at each level of convolution layer of the convolution neural network model, and the second control coefficient set includes second control coefficients corresponding to convolution kernels at each level of convolution layer of the convolution neural network model.

[0116] Step S406: Adjust the control coefficient based on the loss function, and continue the process of adjusting the control coefficient based on the adjusted control coefficient, training data and high-fidelity data until the output of the loss function meets the preset conditions.

[0117] Among them, in the previous step S404, the response of the neural network to the training data and the high-imitation data under the current control coefficient is mined through the loss function E. In this step S406, the control coefficient can be adjusted according to the feedback of the loss function E, including adjusting the first control coefficient and the second control coefficient of the specific convolution kernel. For example, based on the loss function, the first control coefficient and the second control coefficient of each convolution kernel on each convolution layer of the convolution neural network model are adjusted respectively. After adjusting the control coefficient corresponding to the convolution kernel of each convolution layer according to the feedback of the loss function E, the training data and the high-imitation data are input into the adjusted convolution neural network model again to obtain a plurality of training features and a plurality of high-imitation features corresponding to each convolution layer of the adjusted convolution neural network model. In this way, by cyclically generating the training features and the high-imitation features and adjusting the control coefficient, after multiple iterations, it can be achieved that the output of the loss function meets the preset conditions, that is, the optimization effect of the design is achieved. It should be understood that in the process of multiple iterations and adjustments, the training data and the high-imitation data remain unchanged, and other parameters in the model, such as the weights in the convolution layer, are fixed, and only the control coefficient is adjusted to mine redundant convolution kernels. Therefore, the high-imitation data and the high-imitation features generated therefrom remain consistent in terms of the degree of target-related information contained therein. The control coefficient determines the degree of participation of the high-imitation feature in the generation process of subsequent training features. Therefore, the degree of participation of the high-imitation feature can be affected by adjusting the control coefficient, so that the control coefficient of the convolution kernel reflects the influence of the irrelevant factors represented by the high-imitation feature on the specific convolution kernel, and then the importance of the convolution kernel can be measured by comparing the first control coefficient and the second control coefficient.

[0118] Step S408: Based on the control coefficient obtained when the output of the loss function meets the preset conditions, calculate the importance index of each operation operator on each operation layer of the neural network model; and according to the calculated importance index of each operation operator on each operation layer of the neural network model, prune unimportant operation operators to obtain a pruned neural network model.

[0119] Among them, all the control coefficients in the network optimized according to the loss function are obtained in the previous step S406. The first control coefficient can reflect the response of the corresponding convolution kernel to the training data, and the second control coefficient can reflect the response of the corresponding convolution kernel to the high-simulation data. The importance of the convolution kernel can be measured by statistics, refer to formula (7).

[0120]

[0121] Wherein, β^l represents the first control coefficient of the convolution kernel of the Lth convolution layer, β~^l represents the second control coefficient of the convolution kernel of the Lth convolution layer, and s^l represents the metric of the importance of the convolution kernel of the Lth convolution layer. Assume that there are N convolution kernels, each convolution kernel has its own metric, and its own control coefficient must satisfy the constraint condition. As mentioned above, the same convolution kernel generates training features and high-imitation features. If the training features cannot suppress their high-imitation copies, that is, high-imitation features, this is reflected in the first control coefficient being smaller than the corresponding second control coefficient, which means that the convolution kernel is a redundant convolution kernel and should be removed. In this way, the metric calculated by formula (7) can be used to compare with a predetermined threshold. When the metric is greater than the predetermined threshold, it means that the first control coefficient has a sufficiently high proportion relative to the second control coefficient, which means that the corresponding convolution kernel is sufficiently important. Conversely, when the metric is less than the predetermined threshold, it means that the corresponding convolution kernel is not important and should be removed. The predetermined threshold can be set in advance or can be a compression rate given according to a specific application scenario. For example, in scenarios with higher computing power and storage requirements, it may be necessary to set a higher compression rate, so that a higher predetermined threshold can be set to further compress the model and remove redundant convolution kernels.

[0122] In some exemplary embodiments, a step may be added after step S408 to retrain the pruned convolutional neural network model to fine-tune the model parameters to further improve network performance, such as fine-tuning the weight coefficients of the convolution kernels of each convolution layer by retraining.

[0123] Combined with step S400 to step S408, take the operation layer as a convolution layer, the operation operator as a convolution kernel, and the operation as a convolution operation as an example. It should be understood that in addition to the convolution layer and the fully connected layer, the neural network model also has other similar structures that can be called the operation layer of the neural network model, and the operation layer has corresponding operation operators to complete the corresponding operation. For example, there is a convolution kernel on the convolution layer to complete the convolution operation, or there is a fully connected operator on the fully connected layer to complete the fully connected operation, or other structures provide operation operations such as linear superposition. In addition, in the same neural network model, there may be multiple operation layers that have both convolution layers and fully connected layers, and the result of the convolution kernel's convolution operation output can be received by the fully connected operator of the next operation layer and then fully connected. Therefore, in some exemplary embodiments, the convolutional layers mentioned in steps S400 to S408 can be partially replaced by fully connected layers or partially replaced by other operation layers with linear superposition operation components, the convolution kernels can be partially replaced by fully connected operators or partially replaced by other operators or operators that perform linear superposition operations, and the operation operations can correspond to convolution operations, fully connected operations, or other linear superposition operations.

[0124] See also Figure 5 , Figure 5 The flowchart of the method for compressing a network model for image processing provided by an embodiment of the present application is shown. It should be understood that: Figure 5 The network model can be understood as including one or more convolutional layers, one or more fully connected layers, and other possible structures. Among them, the convolutional layer, the fully connected layer and other similar structures can be called the operation layer of the neural network model, and the operation layer has corresponding operation operators to complete the corresponding calculation operations. For example, there is a convolution kernel on the convolution layer to complete the convolution operation, or there is a fully connected operator on the fully connected layer to complete the fully connected operation, or other structures provide calculation operations such as linear superposition. The following takes the operation layer as a convolution layer, the operation operator as a convolution kernel, and the calculation operation as a convolution operation as an example. The following embodiment takes the convolution kernel of the convolution layer of the convolutional neural network model as an example, but it should be understood that it can be applied to neural network models with other operation modes and other architectures. Figure 5 The method shown includes the following steps.

[0125] Step S500: Acquire initial model parameters of the initial neural network model, where the initial model parameters include initial convolution kernels of each convolution layer of the initial neural network model.

[0126] The initial neural network model is a trained network model used in large-scale computer vision applications, such as image classification, target detection, and video analysis. In addition, the initial neural network model is expected to be deployed on edge devices, such as mobile phones and wearable devices. Therefore, due to limited resources and energy consumption, the model must be compressed to meet the requirements of latency and power consumption, while maintaining the prediction accuracy of the model in performing image processing tasks.

[0127] Step S502: Acquire the experimental group images, and obtain the control group images according to the experimental group images and the trained adversarial generative model.

[0128] Among them, the experimental group images refer to a set of images with labels, such as a general training image set for training a neural network model for image recognition, a benchmark data set CIFAR-10, or a large-scale data set ImageNet (ILSVRC-2012). The control group images have a distribution similar to that of the experimental group images, for example, both have a Gaussian distribution. The control group images contain less target-related information than the experimental group images. Here, the target-related information corresponds to information related to the importance of the convolution kernel. The control group images contain as little target-related information as possible, so that irrelevant factors that are not related to the importance of the convolution kernel can be screened out through the control group images, which is conducive to removing redundant convolution kernels by comparing the responses caused by the experimental group images and the control group images respectively.

[0129] Step S504: assign corresponding first control coefficients and second control coefficients to the initial convolution kernels of each convolution layer; input the control group images into the initial neural network model to obtain first image features corresponding to each convolution layer; input the experimental group images into the initial neural network model, and adjust the input of the initial convolution kernels of each convolution layer according to the first control coefficient, the second control coefficient, and the first image features to obtain second image features corresponding to each convolution layer.

[0130] Among them, the first image feature corresponds to the intermediate output result generated by the control group image during the layer-by-layer transmission process in the initial neural network model, and the second image feature corresponds to the intermediate output result generated by the experimental group image during the layer-by-layer transmission process in the initial neural network model. In addition, the generation process of the first image feature is that the first image feature corresponding to the specific convolution layer is obtained by convolving the first image feature corresponding to the previous layer of the specific convolution layer with the convolution kernel of the specific convolution layer. In contrast, the generation process of the second image feature is that the second image feature corresponding to the specific convolution layer is obtained by adjusting the first image feature and the second image feature corresponding to the previous layer of the specific convolution layer according to the first control coefficient and the second control coefficient respectively, and then convolving with the convolution kernel of the specific convolution layer. For a specific convolution kernel, the second image feature output by the previous layer relative to the specific convolution kernel is also the input feature received by the specific convolution kernel in the process of processing the experimental group image transmission. Therefore, the generation process of the second image feature can also be described as: the first image feature corresponding to the specific convolution kernel and the input feature received by the specific convolution kernel are multiplied by the first control coefficient and the second control coefficient respectively, and then the result is added as the input of the adjusted specific convolution kernel. In addition, the first control coefficient and the second control coefficient corresponding to the same convolution kernel meet the constraint condition that the sum is equal to 1.

[0131] Step S506: Calculate the loss function according to the experimental group images, the control group images, the target related information included in the experimental group images, the first control coefficient set, and the second control coefficient set.

[0132] Among them, the first control coefficient set includes the first control coefficient corresponding to each of the initial convolution kernels of the convolution layers at each level, and the second control coefficient set includes the second control coefficient corresponding to each of the initial convolution kernels of the convolution layers at each level. Combined with the generation process of the first image feature mentioned above, the first image feature generated in the network transmission process is consistent with the input control group image in terms of the degree of target-related information contained therein. In other words, the control group image contains as little target-related information as possible, and this property will be maintained during the generation and downward transmission of the first image feature. Therefore, assuming that the control group image does not contain information related to the importance of the convolution kernel, the first image feature does not contain information related to the importance of the convolution kernel. Combined with the generation process of the second image feature mentioned above, the generation of the second image feature involves not only the first image feature output by the previous layer but also the second image feature output by the previous layer, and the respective participation degrees are adjusted respectively by the first control coefficient and the second control coefficient. In addition, the first control coefficient and the second control coefficient must meet the above constraints, which means that when the participation degree of the first image feature is strengthened, it means that the first control coefficient increases, which will inevitably lead to a decrease in the second control coefficient and a weakening of the participation degree of the second image feature. This "one increases while the other decreases" relationship allows us to fully explore the relationship between the network output and the convolution kernel by using the first image feature that contains as little target-related information as possible as a reference and supplement to the second image feature, thereby exploring the redundant convolution kernels in the network model.

[0133] In order to fully exploit the redundant convolution kernels, it is necessary to optimize all the control coefficients in the network through the loss function, so as to fully measure the importance information of each convolution kernel. The input of the loss function is the output of the neural network model, and the output of the neural network model is affected by the experimental group images, the control group images, the first control coefficient set and the second control coefficient set. Therefore, the loss function can use the target-related information included in the experimental group images for supervision, so as to optimize the control coefficients of each convolution kernel. In other words, the loss function can be calculated based on the experimental group images, the control group images, the target-related information included in the experimental group images, the first control coefficient set, and the second control coefficient set.

[0134] Step S508: Determine whether the loss function meets the preset conditions. If so, proceed to step S512; if not, proceed to step S510.

[0135] Among them, by cyclically generating the first image feature and the second image feature and adjusting the control coefficient, after multiple iterations, the output of the loss function can meet the preset conditions, that is, the optimization effect of the design is achieved. It should be understood that during the multiple iterations and adjustments, the control group image and the experimental group image input to the neural network model remain unchanged, that is, the stimulus used to generate the response remains unchanged, and other parameters in the model such as the weights in the convolution layer are fixed, and only the control coefficient is adjusted to test different responses and mine redundant convolution kernels.

[0136] Step S510: adjusting the first control coefficient set and the second control coefficient set based on the loss function.

[0137] When it is determined in the previous step S508 that the preset condition is not met, the control coefficient can be adjusted according to the loss function feedback so as to meet the convergence condition and achieve the optimization purpose.

[0138] Step S512: Based on the first control coefficient set and the second control coefficient set obtained when the output of the loss function meets the preset conditions, select a part of the initial convolution kernels of each level of convolution layer as redundant convolution kernels; cut the selected redundant convolution kernels from the initial neural network model to obtain a compressed neural network model.

[0139] Among them, the same convolution kernel generates the first image feature and the second image feature. If the second image feature based on the experimental group image input cannot suppress its high-imitation copy, that is, the first image feature based on the control group image input, which is reflected in the second control coefficient being smaller than the corresponding first control coefficient, it means that the convolution kernel is a redundant convolution kernel and should be removed. In this way, when the difference between the second control coefficient corresponding to the initial convolution kernel and the first control coefficient corresponding to the initial convolution kernel is less than the preset threshold, the initial convolution kernel is judged to be a redundant convolution kernel. Here, the difference between the second control coefficient and the first control coefficient is a quantitative indicator to measure the importance of the convolution kernel. By comparing the quantitative indicator with the preset threshold, it can be judged whether the convolution kernel is important. The preset threshold can be set in advance or a compression rate given according to a specific application scenario. For example, in a scenario with higher computing power and storage requirements, a higher compression rate may need to be set, so that a higher preset threshold can be set to further compress the model and remove redundant convolution kernels. That is to say, when the difference between the second control coefficient corresponding to the initial convolution kernel and the first control coefficient corresponding to the initial convolution kernel is less than a preset threshold, the initial convolution kernel is judged to be a redundant convolution kernel. In a possible implementation, the first control coefficient set and the second control coefficient set obtained when the output of the loss function meets the preset conditions respectively indicate the irrelevance and importance of the initial convolution kernels of the convolution layers at each level for achieving the goal.

[0140] In some exemplary embodiments, a step may be added after step S512 to retrain the compressed neural network model to fine-tune the model parameters to further improve the network performance, such as fine-tuning the weight coefficients of the convolution kernels of each convolution layer by retraining.

[0141] In a possible implementation, the present application embodiment provides an image processing system arranged on an edge device. The neural network model used by the image processing system for image processing is to convert the uncompressed network model according to Figure 3 or Figure 4 It is obtained by pruning using the method shown.

[0142] In a possible implementation, an embodiment of the present application provides an image processing system. The image processing system includes a processor and a memory, the memory stores computer instructions, and the computer instructions are executed by the processor to perform image processing operations. The image processing operations are based on Figure 4 or Figure 5 The neural network model after compression by the method shown.

[0143] In a possible implementation, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are executed by a processor, the processor executes Figure 3 or Figure 4 The method shown.

[0144] In one possible implementation, Figure 4 or Figure 5 Each level of operation layer of the neural network model involved includes at least one convolution layer, the operation operator of the at least one convolution layer includes at least one convolution kernel, and the operation operation of the at least one convolution kernel is a convolution operation.

[0145] In one possible implementation, Figure 4 or Figure 5 Each level of operation layer of the neural network model involved includes at least one fully connected layer, the operation operator of the at least one fully connected layer includes at least one fully connected operator, and the operation operation of the at least one fully connected operator is a fully connected operation.

[0146] In one possible implementation, Figure 4 or Figure 5 The neural network model involved is a convolutional neural network model, the operation layers at each level are all convolutional layers, the operation operators of the convolutional neural network are all convolution kernels, and the calculation operations of the operation operators of the convolutional neural network are convolution operations.

[0147] In one possible implementation, Figure 4 or Figure 5 The neural network model involved is a fully connected neural network model, the operation layers at all levels are fully connected layers, the operation operators of the fully connected neural network are all fully connected operators, and the calculation operations of the operation operators of the fully connected neural network are fully connected operations.

[0148] See also Figures 1 to 5 The neural network pruned according to the embodiment of the present application is applied to the benchmark dataset CIFAR-10 and the large-scale ImageNet (ILSVRC-2012) dataset. The CIFAR-10 dataset contains 60k RGB images from 10 categories, 50k images for training and 10k for testing. ImageNet (ILSVRC-2012) is a large dataset containing 1.28 million training images and 50k validation images from 1,000 categories.

[0149] Table (1) shows the error of the network pruned by different methods on the test set. Among them, the pruning rate is set to 45%, which achieves a lower test error while reducing more parameters and computation. For example, after pruning 56.0% FLOP of ResNet-56, an error of 6.36% is achieved (the accuracy decreases by 0.06%), which is significantly better than the methods in the prior art (for example, pruning by the HRank method can only achieve an error of 6.83% and 50.0% FLOPs after reduction. Moreover, even for the compact MobileNetV2, the method provided in the embodiment of the present application can still prune 40.3% FLOPs, while the accuracy decreases by only 0.24%.

[0150]

[0151] Table (1) Comparison of different compression methods on the CIFAR-10 dataset

[0152] On the ImageNet (ILSVRC-2012) dataset, the method provided by the embodiment of the present application also surpasses the existing method. Table (2) reports the ImageNet dataset. By pruning the pre-trained network with two different pruning rates, while reducing more computation and parameter amounts, the pruned network maintains a lower test error.

[0153]

[0154] Table (2) Comparison of different compression methods on the ImageNet dataset

[0155] See also Figures 1 to 5, combined with Table (1) and Table (2), the neural network pruning method provided in the embodiment of the present application proposes a technical solution for neural network pruning based on scientific control. Among them, by generating high-simulation features as a control group, the influence of potential irrelevant factors on the pruning process is reduced, and redundant convolution kernels are mined more accurately, thereby obtaining a compact network with high accuracy. Moreover, the method of generating high-simulation features using high-simulation data can efficiently generate high-simulation features for each layer of the network.

[0156] The specific embodiments provided by the present application can be implemented with any one or combination of hardware, software, firmware or solid-state logic circuits, and can be implemented in combination with signal processing, control and / or special circuits. The equipment or device provided by the specific embodiments of the present application may include one or more processors (e.g., microprocessors, controllers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.), which process various computer-executable instructions to control the operation of the equipment or device. The equipment or device provided by the specific embodiments of the present application may include a system bus or a data transmission system that couples various components together. The system bus may include any one or a combination of different bus structures, such as a memory bus or a memory controller, a peripheral bus, a universal serial bus and / or a processor or local bus utilizing any one of a variety of bus architectures. The equipment or device provided by the specific embodiments of the present application may be provided separately, may be a part of a system, or may be a part of other equipment or devices.

[0157] The specific embodiments provided in this application may include a computer-readable storage medium or be combined with a computer-readable storage medium, such as one or more storage devices capable of providing non-temporary data storage. The computer-readable storage medium / storage device may be configured to store data, programmers and / or instructions, which, when executed by a processor of a device or apparatus provided in a specific embodiment of this application, enable these devices or apparatuses to perform related operations. The computer-readable storage medium / storage device may include one or more of the following features: volatility, non-volatility, dynamic, static, readable / writable, read-only, random access, sequential access, location addressability, file addressability, and content addressability. In one or more exemplary embodiments, the computer-readable storage medium / storage device may be integrated into the device or apparatus provided in a specific embodiment of this application or belong to a common system. Computer-readable storage media / storage devices may include optical storage devices, semiconductor storage devices and / or magnetic storage devices, etc., and may also include random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, recordable and / or rewritable compact disks (CDs), digital versatile disks (DVDs), mass storage media devices or any other form of suitable storage media.

[0158] The above is the implementation method of the embodiment of the present application. It should be noted that the steps in the method described in the specific embodiment of the present application can be adjusted in order, merged and deleted according to actual needs. In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, refer to the relevant description of other embodiments. It is understandable that the structure shown in the embodiment of the present application and the accompanying drawings does not constitute a specific limitation on the relevant device or system. In other embodiments of the present application, the relevant device or system may include more or fewer components than the specific embodiments and the accompanying drawings, or combine certain components, or split certain components, or have different component arrangements. Those skilled in the art will understand that, without departing from the spirit and scope of the specific embodiments of the present application, various modifications or changes can be made to the arrangement, operation and details of the methods and equipment recorded in the specific embodiments; without departing from the principles of the embodiments of the present application, several improvements and modifications can also be made, and these improvements and modifications are also regarded as the scope of protection of the present application.

Claims

1. A method for pruning a neural network model, characterized in that: The method comprises: Get labeled training data; Acquire high-imitation data based on the training data, wherein the high-imitation data has a similar distribution pattern to the training data, and the high-imitation data does not contain information related to the label; Inputting the training data and the high-simulation data into the neural network model to obtain a plurality of training features and a plurality of high-simulation features corresponding to each level of operation layers of the neural network model, respectively, wherein the operation operators on each level of operation layers of the neural network model are respectively assigned control coefficients; According to the training data, the high-simulation data, the labels and the control coefficients, unimportant operation operators are pruned to obtain a pruned neural network model.

2. The method according to claim 1, characterized in that The method of pruning unimportant operation operators according to the training data, the high-simulation data, the labels and the control coefficients to obtain a pruned neural network model includes: Calculate a loss function according to the training data, the high-simulation data, the label and the control coefficient; Based on the loss function, adjusting the control coefficient; continuing the process of adjusting the control coefficient until the output of the loss function meets a preset condition; Based on the control coefficient obtained when the output of the loss function meets the preset conditions, calculating the importance index of each operation operator on each level of the operation layer of the neural network model; and According to the calculated importance index of each operation operator on each operation layer of the neural network model, unimportant operation operators are pruned to obtain a pruned neural network model.

3. The method according to claim 2, characterized in that The training features corresponding to the specific operation layer are determined based on the high imitation features and training features corresponding to the previous layer of the specific operation layer and the control coefficient corresponding to the operation operator of the specific operation layer; the high imitation features corresponding to the specific operation layer are obtained by performing calculations on the high imitation features corresponding to the previous layer of the specific operation layer and the operation operator of the specific operation layer.

4. The method according to claim 3, characterized in that The control coefficient includes a first control coefficient and a second control coefficient, and the training feature corresponding to the specific operation layer is determined according to the high imitation feature and the training feature corresponding to the previous layer of the specific operation layer and the control coefficient corresponding to the operation operator of the specific operation layer, including: The training features corresponding to the specific operation layer are obtained by adjusting the training features and high-imitation features corresponding to the previous layer of the specific operation layer according to the first control coefficient and the second control coefficient, respectively, adding them together, and then performing an operation operation with the operation operator of the specific operation layer.

5. The method according to claim 4, characterized in that The first control coefficient and the second control coefficient corresponding to the same operator satisfy the constraint condition that the sum of the coefficients added together is equal to 1.

6. The method according to claim 5, characterized in that Calculating the loss function according to the training data, the high-simulation data, the label and the control coefficient includes: The loss function is calculated according to the training data, the high-simulation data, the label, a first control coefficient set, and a second control coefficient set, wherein the first control coefficient set includes first control coefficients corresponding to the operation operators on each level of the operation layer of the neural network model, and the second control coefficient set includes second control coefficients corresponding to the operation operators on each level of the operation layer of the neural network model.

7. The method according to claim 6, characterized in that Based on the loss function, adjusting the control coefficient includes: Based on the loss function, the first control coefficient and the second control coefficient of each operation operator on each level of the operation layer of the neural network model are adjusted respectively.

8. The method according to claim 7, characterized in that Based on the control coefficient obtained when the output of the loss function meets the preset conditions, the importance index of each operation operator on each operation layer of the neural network model is calculated, including: For each operation operator of the neural network model, a difference between a first control coefficient corresponding to the operation operator and a second control coefficient corresponding to the operation operator is calculated as an importance index corresponding to the operation operator.

9. The method according to claim 8, characterized in that According to the calculated importance indexes of the operation operators at each level of the operation layer of the neural network model, unimportant operation operators are pruned to obtain a pruned neural network model, including: determining a significance threshold based on a predetermined compression rate for the neural network model; The importance index of each operation operator of the neural network model is compared with the importance threshold, and when the importance index is less than the importance threshold, it is judged as an unimportant operation operator.

10. The method according to any one of claims 1 to 9, characterized in that The operation layers at each level include at least one convolution layer, the operation operator of the at least one convolution layer includes at least one convolution kernel, and the operation operation of the at least one convolution kernel is a convolution operation.

11. The method according to any one of claims 1 to 9, characterized in that The operation layers at each level include at least one fully connected layer, the operation operator of the at least one fully connected layer includes at least one fully connected operator, and the operation operation of the at least one fully connected operator is a fully connected operation.

12. The method according to any one of claims 1 to 9, characterized in that The neural network model is a convolutional neural network model, the operation layers at each level are all convolutional layers, the operation operators of the convolutional neural network are all convolution kernels, and the calculation operations of the operation operators of the convolutional neural network are convolution operations.

13. The method according to any one of claims 1 to 9, characterized in that The neural network model is a fully connected neural network model, the operation layers at all levels are fully connected layers, the operation operators of the fully connected neural network are all fully connected operators, and the calculation operations of the operation operators of the fully connected neural network are fully connected operations.

14. The method according to any one of claims 1 to 9, characterized in that The training data is a training image, the high-imitation data is a high-imitation image obtained by inputting the training image into a generative adversarial network model, and the neural network model is used for image processing, wherein the training data and the high-imitation data are input into the neural network model to obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layer of the neural network model, respectively, including: The training image and the high-imitation image are input into the neural network model to obtain a plurality of training image features and a plurality of high-imitation image features corresponding to each level of operation layers of the neural network model.

15. An image processing system arranged on an edge device, characterized in that: The neural network model used by the image processing system for image processing is obtained by pruning an uncompressed network model according to the method described in claim 14.

16. The method according to any one of claims 1 to 9, characterized in that The training data is a training speech signal, and the high-imitation data is a high-imitation speech signal obtained by inputting the training speech signal into a generative adversarial network model, wherein the neural network model is used for speech processing, wherein the training data and the high-imitation data are input into the neural network model to obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layer of the neural network model, respectively, including: The training voice signal and the high-imitation voice signal are input into the neural network model to obtain a plurality of training voice signal features and a plurality of high-imitation voice signal features corresponding to each level of operation of the neural network model.

17. A speech processing system arranged on an edge device, characterized in that: The neural network model used by the speech processing system for speech signal processing is obtained by pruning the uncompressed network model according to the method described in claim 15.

18. The method according to any one of claims 1 to 9, characterized in that The training data is a training natural language signal, and the high-imitation data is a high-imitation natural language signal obtained by inputting the training natural language signal into a generative adversarial network model, wherein the neural network model is used for natural language processing, wherein the training data and the high-imitation data are input into the neural network model to obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layer of the neural network model, respectively, including: The training natural language signal and the high-simulation natural language signal are input into the neural network model to obtain a plurality of training natural language signal features and a plurality of high-simulation natural language signal features corresponding to each level of operation of the neural network model.

19. A natural language processing system deployed on an edge device, characterized in that: The neural network model used by the natural language processing system for natural language signal processing is obtained by pruning an uncompressed network model according to the method described in claim 18.

20. The method according to any one of claims 1 to 9, characterized in that The training data is a training recommendation content signal, and the high-imitation data is a high-imitation recommendation content signal obtained by inputting the training recommendation content signal into a generative adversarial network model, wherein the neural network model is used to screen the recommended content, wherein the training data and the high-imitation data are input into the neural network model to obtain a plurality of training features and a plurality of high-imitation features corresponding to each level of operation layer of the neural network model, respectively, including: The training recommended content signal and the high-imitation recommended content signal are input into the neural network model to obtain a plurality of training recommended content signal features and a plurality of high-imitation recommended content signal features corresponding to each level of operation of the neural network model.

21. A content recommendation system deployed on an edge device, characterized in that: The neural network model used by the content recommendation system to screen recommended content is obtained by pruning an uncompressed network model according to the method described in claim 20.

22. A method for compressing a neural network model, characterized in that: The method comprises: Acquire initial model parameters of the initial neural network model, wherein the initial model parameters include initial convolution kernels of each convolution layer of the initial neural network model; Obtaining a control group of images according to the experimental group of images and the trained adversarial generative model, wherein the control group of images has a distribution similar to that of the experimental group of images, and the control group of images contains less target-related information than the experimental group of images; Based on the experimental group images and the control group images, a part of the initial convolution kernels of the convolution layers at each level is selected as redundant convolution kernels; the selected redundant convolution kernels are cut off from the initial neural network model to obtain a compressed neural network model.

23. The method according to claim 22, characterized in that The selecting, based on the experimental group image and the control group image, a part of the initial convolution kernels of the convolution layers at each level as redundant convolution kernels comprises: Inputting the control group image into the initial neural network model to obtain first image features corresponding to each level of convolutional layers; Allocating corresponding first control coefficients and second control coefficients to initial convolution kernels of each level of convolution layers; Inputting the experimental group image into the initial neural network model, and adjusting the input of the initial convolution kernel of each convolution layer according to the first control coefficient, the second control coefficient, and the first image feature to obtain second image features corresponding to each convolution layer one by one; Calculate a loss function according to the experimental group image, the control group image, target related information included in the experimental group image, a first control coefficient set, and a second control coefficient set; the first control coefficient set includes first control coefficients corresponding to the initial convolution kernels of each level of convolution layer, and the second control coefficient set includes second control coefficients corresponding to the initial convolution kernels of each level of convolution layer; Based on the loss function, adjusting the first control coefficient set and the second control coefficient set; continuing the adjustment process until the output of the loss function meets a preset condition; Based on the first control coefficient set and the second control coefficient set obtained when the output of the loss function meets the preset conditions, a part of the initial convolution kernels of the convolution layers at each level is selected as the redundant convolution kernels.

24. The method according to claim 23, characterized in that According to the first control coefficient, the second control coefficient, and the first image feature, adjusting the input of the initial convolution kernel of each level of convolution layer to obtain the second image feature corresponding to each level of convolution layer in a one-to-one manner, including: The first image feature corresponding to the initial convolution kernel and the input feature received by the initial convolution kernel are respectively multiplied by the first control coefficient and the second control coefficient and then added together to obtain the result as the input of the adjusted initial convolution kernel.

25. The method according to claim 24, characterized in that The first control coefficient and the second control coefficient corresponding to the same convolution kernel satisfy the constraint that their sum is equal to 1.

26. The method according to claim 25, characterized in that Based on the first control coefficient set and the second control coefficient set obtained when the loss value satisfies a preset condition, selecting a part of the initial convolution kernels of the convolution layers at each level as redundant convolution kernels, including: When the difference between the second control coefficient corresponding to the initial convolution kernel and the first control coefficient corresponding to the initial convolution kernel is less than a preset threshold, it is determined that the initial convolution kernel is a redundant convolution kernel.

27. The method according to claim 25, characterized in that The first control coefficient set and the second control coefficient set obtained when the output of the loss value meets the preset conditions respectively indicate the irrelevance and importance of the initial convolution kernels of the convolution layers at each level for achieving the goal.

28. An image processing system, comprising a processor and a memory, wherein the memory stores computer instructions, and the computer instructions are executed by the processor to perform image processing operations, characterized in that: The image processing operation is performed according to the neural network model compressed by the method according to any one of claims 24-27.

29. A computer storage medium storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the processor performs the following operations: Get labeled training data; Acquire high-imitation data based on the training data, wherein the high-imitation data has a similar distribution pattern to the training data, and the high-imitation data does not contain information related to the label; Inputting the training data and the high-simulation data into the neural network model to obtain a plurality of training features and a plurality of high-simulation features corresponding to each level of operation layers of the neural network model, respectively, wherein the operation operators on each level of operation layers of the neural network model are respectively assigned control coefficients; According to the training data, the high-simulation data, the labels and the control coefficients, unimportant operation operators are pruned to obtain a pruned neural network model.

30. The computer storage medium of claim 29, wherein: The method of pruning unimportant operation operators according to the training data, the high-simulation data, the labels and the control coefficients to obtain a pruned neural network model includes: Calculate a loss function according to the training data, the high-simulation data, the label and the control coefficient; Based on the loss function, adjusting the control coefficient; continuing the process of adjusting the control coefficient until the output of the loss function meets a preset condition; Based on the control coefficient obtained when the output of the loss function meets the preset conditions, calculating the importance index of each operation operator on each level of the operation layer of the neural network model; and According to the calculated importance index of each operation operator on each operation layer of the neural network model, unimportant operation operators are pruned to obtain a pruned neural network model.

31. The computer storage medium of claim 30, wherein: The training features corresponding to the specific operation layer are determined according to the high imitation features and training features corresponding to the previous layer of the specific operation layer and the control coefficient corresponding to the operation operator of the specific operation layer.

Images