Image classification method and device based on pruning neural network, equipment and medium

Abstract

The invention discloses an image classification method and device based on a pruning neural network, equipment and a medium, and the method comprises the steps: traversing a convolution layer of an original image classification model, and constructing a Gaussian mixture model based on the weight data of the convolution layer; calculating a zero activation rate of a convolution kernel in each convolution layer; based on the Gaussian mixture model and the zero activation rate of the convolution kernels, calculating the composite convolution correlation index of each convolution kernel, selecting the convolution layer with the highest average value of the composite convolution correlation indexes as a target convolution layer, removing the convolution kernels with the composite convolution correlation indexes greater than a preset threshold in the target convolution layer, and obtaining the target convolution layer. A pruned image classification model is obtained; and inputting image data to be processed into the pruned image classification model to obtain image classification data. According to the method provided by the invention, the image classification model can be compressed while the model precision is ensured, so that the feasibility of deploying the image classification model on lightweight equipment such as edge side nodes is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image classification method, device, equipment and medium based on a pruned neural network. Background Art

[0002] At present, convolutional neural networks are often used for image classification. Convolutional neural network models have high accuracy and ideal effects in visual tasks, but they also have disadvantages such as complex structure, high storage overhead, and high computing resources. However, due to the limited storage and computing resources of edge node devices, existing image classification models are difficult to accurately and efficiently implement on edge hardware platforms.

[0003] Existing solutions achieve model compression by pruning the model. However, the norm of the convolution kernel weight value is usually used to determine whether to prune. The effect is poor. Studies have shown that clustering methods are more suitable for model structured pruning. The K-means clustering method, which is commonly used in model pruning research, assumes that the target data is symmetrically distributed about the cluster center circle. In the face of convolutional layer weights with complex data distribution, it cannot well characterize the distribution of data. In addition, each data point in the K-means clustering method can only be assigned to one category cluster, and it cannot handle the fuzzy boundaries or overlaps in the data well. Summary of the invention

[0004] The embodiments of the present application provide an image classification method, apparatus, device and storage medium based on a pruned neural network, so as to at least solve the technical problem in the related art that it is difficult to efficiently prune the image classification convolution model.

[0005] According to one aspect of an embodiment of the present application, there is provided an image classification method based on a pruned neural network, comprising:

[0006] Traverse the convolution layers of the original image classification model, and build a Gaussian mixture model based on the weight data of the convolution layers; calculate the zero activation rate of the convolution kernel in each convolution layer; calculate the composite convolution association index of each convolution kernel based on the Gaussian mixture model and the zero activation rate of the convolution kernel, wherein the composite convolution association index is obtained by the zero activation rate, the similarity and the overall distribution influence; and count the mean of the composite convolution association index of each convolution layer; select the convolution layer with the highest mean of the composite convolution association index as the target convolution layer, remove the convolution kernel whose composite convolution association index is greater than a preset threshold in the target convolution layer, and obtain the pruned image classification model; input the image data to be processed into the pruned image classification model to obtain image classification data.

[0007] In one embodiment, constructing a Gaussian mixture model based on the weight data of the convolutional layer includes:

[0008] Extract the weight data of the convolutional layer;

[0009] Constructing a Gaussian mixture model according to the weight data;

[0010] The parameters of the Gaussian mixture model are solved based on the expectation maximization algorithm to obtain the mean, variance and probability of each Gaussian distribution obeying the weight of each Gaussian distribution.

[0011] In one embodiment, the composite convolution correlation index of each convolution kernel is calculated based on the Gaussian mixture model and the zero activation rate of the convolution kernel, including:

[0012] Based on the Gaussian mixture model, calculating the similarity of each convolution kernel;

[0013] Based on the Gaussian mixture model, the overall distribution influence of each convolution kernel is calculated;

[0014] Based on the zero activation rate, the similarity and the sum of the overall distribution influence of each convolution kernel, the composite convolution association index of each convolution kernel is obtained.

[0015] In one embodiment, based on the Gaussian mixture model, calculating the similarity of each convolution kernel includes:

[0016] Based on the Gaussian mixture model, the probability that the weight data of the convolution kernel obeys each Gaussian distribution is obtained;

[0017] Based on the probability that all weight data in the convolution kernel obey a certain Gaussian distribution and the ratio of the number of weights of the convolution kernel, the membership of the convolution kernel to the Gaussian distribution is obtained;

[0018] Based on the membership of all Gaussian distributions of the convolution kernel, a membership set is obtained;

[0019] Constructing a vector according to the membership set of the convolution kernels, and using the vector to calculate the cosine similarity between the convolution kernels;

[0020] Based on the cosine similarity between the convolution kernels, the similarity of each convolution kernel is determined.

[0021] In one embodiment, based on the Gaussian mixture model, calculating the overall distribution influence of each convolution kernel includes:

[0022] Calculating the variance of the convolution kernel based on the membership set of the convolution kernel;

[0023] The overall distribution influence of the convolution kernel is determined according to the variance.

[0024] In one embodiment, after removing the convolution kernel whose composite convolution correlation index is greater than a preset threshold, the method further includes:

[0025] Calculate the mean of the composite convolutional correlation index for each convolutional layer in the network;

[0026] The convolution kernel of the convolution layer with the highest mean value of the composite convolution correlation index is selected for pruning until the mean value of the composite convolution correlation index of each convolution layer is lower than the preset threshold, or until the pruning rate is greater than the preset threshold, and then stop pruning.

[0027] In one embodiment, after stopping pruning, the method further includes:

[0028] Retrain and fine-tune the parameters of the pruned image classification model.

[0029] According to one aspect of an embodiment of the present application, there is also provided an image classification device based on a pruned neural network, comprising:

[0030] A construction module, used for traversing the convolutional layers of the original image classification model, and constructing a Gaussian mixture model based on the weight data of the convolutional layers;

[0031] A zero activation rate calculation module is used to calculate the zero activation rate of the convolution kernel in each convolution layer; a correlation index calculation module is used to calculate the composite convolution correlation index of each convolution kernel based on the Gaussian mixture model and the zero activation rate of the convolution kernel, wherein the composite convolution correlation index is obtained by the zero activation rate, similarity and overall distribution influence degree; and the mean value of the composite convolution correlation index of each convolution layer is calculated;

[0032] A pruning module is used to select the convolution layer with the highest mean value of the composite convolution correlation index as the target convolution layer, remove the convolution kernels whose composite convolution correlation index is greater than a preset threshold in the target convolution layer, and obtain a pruned image classification model;

[0033] The image classification module is used to input the image data to be processed into the pruned image classification model to obtain image classification data.

[0034] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned image classification method based on pruned neural network through the computer program.

[0035] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned image classification method based on pruned neural network when running.

[0036] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0037] The solution provided in the embodiment of the present application uses zero activation rate to evaluate the importance of convolution kernels, and uses Gaussian mixture model to analyze the distribution of convolution layer weights in convolutional neural networks. It is considered that the weight distribution is the superposition of several Gaussian distributions, and the probability of each data point belonging to each cluster is given, rather than just assigning it to a single cluster, so as to improve the applicability of lightweight algorithms when there are fuzzy boundaries or overlaps in the data set. Based on the Gaussian distribution, the similarity of the convolution kernel and the overall distribution influence are evaluated, and the composite convolution association index is obtained by combining the zero activation rate, similarity and overall distribution influence. The convolution kernels with a composite convolution association index greater than the preset threshold are removed to obtain a pruned image classification model, which comprehensively considers the importance, similarity and overall distribution influence of the convolution kernel. After being deployed on the edge computing platform, the image processing model pruned using this algorithm has a faster reasoning speed, higher model accuracy, and less precision loss, and has good engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 is a flow chart of an image classification method based on a pruned neural network according to an embodiment of the present application;

[0040] Figure 2 is a flowchart of another image classification method based on a pruned neural network according to an embodiment of the present application;

[0041] Figure 3 is a schematic diagram of another image classification method based on a pruned neural network according to an embodiment of the present application;

[0042] Figure 4 is a schematic diagram of a Gaussian mixture model for constructing weighted data according to an embodiment of the present application;

[0043] Figure 5 is a schematic diagram of solving parameters according to an EM algorithm according to an embodiment of the present application;

[0044] Figure 6 is a schematic diagram of algorithm iteration and parameter fine-tuning according to an embodiment of the present application;

[0045] Figure 7 is a schematic diagram of an image classification device based on a pruned neural network according to an embodiment of the present application;

[0046] Figure 8 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] This application uses a Gaussian mixture model to analyze the distribution of convolutional layer weights in a convolutional neural network. It is considered that the weight distribution is the superposition of several Gaussian distributions, and the probability that each weight data obeys a certain Gaussian distribution is given. The similarity between the convolution kernels and the impact on the overall distribution are evaluated based on the Gaussian distribution measurement results of the weight data in the convolution kernel. The APoZ (zero activation rate) value of the convolution kernel is calculated to evaluate its importance, and then pruning is performed based on the similarity, importance and overall distribution impact of the convolution kernel.

[0050] The following is combined with Figure 1 The image classification method based on the pruned neural network in the embodiment of the present application is described in detail. Figure 1 As shown, the method mainly includes the following steps:

[0051] S101 traverses the convolutional layer of the original image classification model and builds a Gaussian mixture model based on the weight data of the convolutional layer.

[0052] The core task of the image classification model is to output the corresponding category of the input image, and the specific output value accuracy requirement of the model is not high. However, the output category accuracy loss of the trained image classification model is low after pruning. Therefore, this paper designs a convolution kernel pruning algorithm based on Gaussian mixture model and zero activation rate value for the trained image classification model. Resnet18 is a typical network model used for image classification.

[0053] In one embodiment, the data set uses the cifar-10 data set, taking the resnet18 convolutional neural network image classification model as an example.

[0054] Specifically, the convolutional layers in the model are traversed, and a Gaussian mixture model is constructed based on the weight data of the convolutional layers. The Gaussian mixture model refers to the situation where the data obeys the superposition of multiple Gaussian distributions, usually referring to a random variable or a set of data points that is a mixture of several different normal distributions. This is called a mixed normal distribution or a Gaussian mixture model in statistics.

[0055] Specifically, the weight data of the convolutional layer is extracted; a Gaussian mixture model is constructed based on the weight data; the parameters of the Gaussian mixture model are solved based on the expectation-maximization algorithm to obtain the mean and variance of each Gaussian distribution and the probability that the weight obeys each Gaussian distribution.

[0056] like Figure 4 As shown, for the pruning target convolution layer weight X = {x 1 ,x 2 ,...,x n} as observation data, n is the number of weights of the convolutional layer. Assuming that the weight data obeys the superposition of K Gaussian distributions, the GMM (Gaussian Mixture Model) modeling diagram is as follows: Figure 4 shown.

[0057] The probability density function of the weighted data is:

[0058]

[0059] Among them, p i represents the probability that the weight x follows the i-th Gaussian distribution, N(x|μ i ,∑ i ) represents the probability density function of the i-th Gaussian distribution, μ i and∑ i Represents the mean and variance of the Gaussian distribution. After completing the GMM modeling of the weighted data, it is necessary to solve the unknown parameter θ in the formula:

[0060] θ={p 1 ,p 2 ,...,p K ,μ 1 ,μ 2 ,...,μ K ,∑ 1 ,∑ 2 ,...∑ K};

[0061] Furthermore, the maximum likelihood estimation is used to determine the unknown parameter θ, and the maximum likelihood function L(θ|X) is:

[0062]

[0063] There are many weights in the convolutional neural network. If the above formula is directly derived, it is difficult to obtain an analytical solution to the parameters. This application uses the EM (expectation maximization) algorithm to solve the Gaussian mixture model parameters of the weight data.

[0064] Use the EM algorithm to solve the parameters after GMM modeling. The process is as follows Figure 5 As shown, it includes obtaining target data, randomly selecting initial parameter values, performing step E, estimating the probability that the data point belongs to each component based on the current parameters; performing step M, re-estimating the parameters of the component based on the probability obtained in step E, and by continuously iterating these two steps, the converged model parameters can be finally obtained.

[0065] The unknown parameters are solved by the EM algorithm to obtain the mean, variance and probability of each Gaussian distribution obeying the weight.

[0066] S102 calculates the zero activation rate of the convolution kernel in each convolution layer.

[0067] Specifically, the APoZ (Average Percentage of Zeros) value of the network convolution kernel is calculated. The APoZ calculation formula of the convolution kernel is:

[0068]

[0069] in, represents the APoZ of the cth channel of the i-th layer, represents the value of the jth element in the output of the cth channel of the i-th layer. If f(·) is 1, otherwise it is 0. N represents the number of samples for measuring APoZ, and M represents The dimension of the output feature map. The larger the APoZ of the convolution kernel, the greater the proportion of zero output, and the less important the convolution kernel is to the neural network.

[0070] S103 calculates the composite convolution association index of each convolution kernel based on the Gaussian mixture model and the zero activation rate of the convolution kernel, where the composite convolution association index is obtained by the zero activation rate, similarity, and overall distribution influence; and calculates the mean composite convolution association index of each convolution layer.

[0071] Specifically, first obtain the zero activation rate of each convolution kernel to obtain the APoZ value of each convolution kernel.

[0072] Furthermore, based on the Gaussian mixture model, the similarity of each convolution kernel is calculated.

[0073] In one embodiment, based on a Gaussian mixture model, the probability that the weight data of the convolution kernel obeys each Gaussian distribution is obtained; based on the ratio of the probability that all weight data in the convolution kernel obey a certain Gaussian distribution to the number of weights of the convolution kernel, the membership of the convolution kernel to the Gaussian distribution is obtained; based on the membership of the convolution kernel to all Gaussian distributions, a membership set is obtained.

[0074] Assume that the convolutional layer consists of {k 1 ,k 2 ,...,k m}There are m convolution kernels in total, and the jth convolution kernel has Total M j After the weight data is modeled by GMM, it satisfies the superposition of K Gaussian distributions. The tth weight data The probability of obeying the i-th Gaussian distribution is Then there is

[0075] Furthermore, based on the ratio of the probability that all weight data in the convolution kernel obey a certain Gaussian distribution and the number of weights of the convolution kernel, the membership of the convolution kernel to the Gaussian distribution is obtained, that is, the degree to which the convolution kernel obeys the Gaussian distribution.

[0076]

[0077] in, Represents the jth convolution kernel for the i-th Gaussian distribution c i The membership of the jth convolution kernel to all Gaussian distributions is

[0078] Furthermore, a vector is constructed according to the membership set of the convolution kernel. Each convolution kernel has a membership vector. If the membership of two convolution kernels to the Gaussian distribution is similar, the vectors of the two convolution kernels are also similar.

[0079] The cosine similarity between convolution kernels is calculated using vectors, and the similarity of each convolution kernel is determined based on the cosine similarity between the convolution kernels. For example, if there are 10 convolution kernels and the cosine similarity is calculated for each of them, then for one of the convolution kernels, there are nine similarity values ​​with the other nine convolution kernels. These nine similarity values ​​are added together to get the similarity of the convolution kernel. Because the membership is positive, there will not be a situation where the cosine similarity of the vectors constructed by two convolution kernels is negative. In this case, the value of the cosine similarity is 0 to 1, 1 means complete overlap, and 0 means mutual independence. The larger the cosine similarity, the higher the similarity between the convolution kernel and other convolution kernels.

[0080] Furthermore, based on the Gaussian mixture model, the overall distribution influence of each convolution kernel is calculated, including calculating the variance of the convolution kernel based on the membership set of the convolution kernel, and determining the overall distribution influence of the convolution kernel according to the variance.

[0081] In one embodiment, the membership set of the jth convolution kernel to the Gaussian distribution is obtained as And calculate the variance s(Q j ). When the membership variance of a convolution kernel is large, it means that the convolution kernel is only highly compliant with the superposition of several Gaussian distributions. Such a convolution kernel focuses more on extracting local features. The larger the variance, the greater the influence of the convolution kernel on the overall distribution. When the membership variance of a convolution kernel is small, it means that the convolution kernel is more compliant with the overall distribution of the data. When pruning, such convolution kernels are retained first and convolution kernels with large variances are removed.

[0082] Finally, based on the zero activation rate, similarity, and overall distribution influence of each convolution kernel, the composite convolution correlation index of each convolution kernel is obtained.

[0083] In another optional implementation, a first APoZ-S value may be obtained according to the sum of the zero activation rate and the similarity of each convolution kernel, and pruning may be performed according to the first APoZ-S value being greater than a threshold value, in which case only the zero activation rate APoZ and the similarity similarity are considered. Alternatively, a second APoZ-S value may be obtained according to the sum of the zero activation rate and the overall distribution influence of each convolution kernel, and pruning may be performed according to the second APoZ-S value being greater than a threshold value, in which case only the zero activation rate APoZ and the overall distribution influence (variance s(Q j )).

[0084] According to this step, during pruning, not only the importance of the convolution kernel is considered, but also the similarity between the convolution kernels and the overall data compliance are considered in combination with the compliance of the Gaussian distribution to ensure the efficiency, compression rate and accuracy of the pruned model.

[0085] S104 selects the convolution layer with the highest mean value of the composite convolution correlation index as the target convolution layer, removes the convolution kernels with composite convolution correlation index greater than a preset threshold in the target convolution layer, and obtains a pruned image classification model.

[0086] The target convolution layer is the layer to be pruned and is also the convolution layer with the highest mean value of the composite convolution correlation index. The composite convolution correlation index of each convolution kernel in the target convolution layer is calculated, and the convolution kernels are sorted based on the composite convolution correlation index. The appropriate importance threshold T is set according to the required pruning ratio. When the composite convolution correlation index of the convolution kernel is greater than T, the convolution kernel is pruned, otherwise the convolution kernel is retained.

[0087] In one embodiment, after removing the convolution kernel whose composite convolution correlation index is greater than a preset threshold, the method further includes:

[0088] Calculate the mean of the composite convolution correlation index of each convolution layer in the network; select the convolution kernel of the convolution layer with the highest composite convolution correlation index mean for pruning, until the mean of the composite convolution correlation index of each convolution layer is lower than the preset threshold, or until the pruning rate is greater than the preset threshold, stop pruning. Among them, the threshold and pruning rate provided in the embodiment of the present application can be set according to actual needs, and this application does not make specific restrictions.

[0089] In one embodiment, after stopping pruning, the method further includes: retraining and fine-tuning parameters of the pruned image classification model. Figure 6 As shown in the figure, it is difficult to obtain a network with high compression rate and high precision by single pruning and retraining based on the composite convolutional relevance index value. The algorithm is iterated. First, the composite convolutional relevance index (CCRI) value of the convolution kernel is calculated, and the convolution layer with the highest CCRI mean is selected. The convolution kernel of this layer is pruned until the preset number of iterations is reached. The pruned model is retrained and parameter fine-tuned, and the pruned model is output.

[0090] S105 inputs the image data to be processed into the pruned image classification model to obtain image classification data.

[0091] Image classification is performed according to the pruned image classification model. The pruned image classification model can be deployed on an edge device with less computing resources and smaller storage space, so that image data can be classified.

[0092] In order to facilitate understanding of the method provided in the embodiment of the present application, Figure 2 Further description, such as Figure 2 As shown, the method includes:

[0093] Get the original model, calculate the APoZ value of the convolution kernel in the model, and perform GMM modeling on the convolution layer weight data. Calculate the CCRI value of the convolution kernel, and count the CCRI mean of each convolution layer. Take the convolution layer with the highest CCRI mean as the target convolution layer, sort the importance according to the CCRI value, and remove the convolution kernels whose CCRI values ​​of the target convolution layer are greater than the preset threshold.

[0094] Furthermore, algorithm iteration and parameter fine-tuning are performed to output the pruned model.

[0095] Figure 3 Another implementation method is shown below. Figure 3 Further description, such as Figure 3As shown, the method includes:

[0096] Calculate the APoZ value of the convolution kernel in the model based on the sample data, and then count the APoZ mean of the convolution kernel in each convolution layer. The convolution layer with the highest mean is used as the target convolution layer.

[0097] Furthermore, the weight data of the m convolution kernels of the target convolution layer are modeled using GMM. The membership variance of the convolution kernel is calculated, and the APoZ-S value of the convolution kernel is obtained based on the sum of the membership variance and the zero activation rate. The importance is sorted according to the APoZ-S value, and the convolution kernels with APoZ-S values ​​greater than the preset threshold are removed to obtain the pruned convolution layer.

[0098] According to another aspect of the embodiment of the present application, there is also provided an image classification device based on a pruned neural network for implementing the above-mentioned image classification method based on a pruned neural network. Figure 7 As shown, the device comprises:

[0099] A construction module 701 is used to traverse the convolutional layer of the original image classification model and construct a Gaussian mixture model based on the weight data of the convolutional layer;

[0100] A zero activation rate calculation module 702, used to calculate the zero activation rate of the convolution kernel in each convolution layer;

[0101] The association index calculation module 703 is used to calculate the composite convolution association index of each convolution kernel based on the Gaussian mixture model and the zero activation rate of the convolution kernel, where the composite convolution association index is obtained by the zero activation rate, the similarity and the overall distribution influence degree; and to calculate the mean value of the composite convolution association index of each convolution layer;

[0102] A pruning module 704 is used to select a convolution layer with the highest mean value of the composite convolution correlation index as a target convolution layer, and remove convolution kernels with a composite convolution correlation index greater than a preset threshold in the target convolution layer to obtain a pruned image classification model;

[0103] The image classification module 705 is used to input the image data to be processed into the pruned image classification model to obtain image classification data.

[0104] It should be noted that the image classification device based on pruned neural network provided in the above embodiment only uses the division of the above functional modules as an example when executing the image classification method based on pruned neural network. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the image classification device based on pruned neural network provided in the above embodiment and the image classification method based on pruned neural network embodiment belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0105] According to another aspect of the embodiments of the present application, an electronic device corresponding to the image classification method based on pruned neural network provided in the aforementioned embodiments is also provided to execute the above-mentioned image classification method based on pruned neural network.

[0106] Please refer to Figure 8 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 8 As shown, the electronic device includes: a processor 800, a memory 801, a bus 802 and a communication interface 803, and the processor 800, the communication interface 803 and the memory 801 are connected via the bus 802; the memory 801 stores a computer program that can be run on the processor 800, and when the processor 800 runs the computer program, the image classification method based on the pruned neural network provided in any of the aforementioned embodiments of the present application is executed.

[0107] The memory 801 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 803 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0108] The bus 802 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs, and the processor 800 executes the programs after receiving the execution instructions. The image classification method based on the pruned neural network disclosed in any of the embodiments of the present application may be applied to the processor 800, or implemented by the processor 800.

[0109] The processor 800 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 800. The above processor 800 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 801, and the processor 800 reads the information in the memory 801 and completes the steps of the above method in combination with its hardware.

[0110] The electronic device provided in the embodiment of the present application and the image classification method based on pruned neural network provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0111] According to another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium corresponding to the image classification method based on pruned neural network provided in the aforementioned embodiments, on which a computer program (i.e., program product) is stored. When the computer program is executed by a processor, the image classification method based on pruned neural network provided in any of the aforementioned embodiments is executed.

[0112] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0113] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the image classification method based on pruned neural network provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0114] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the attached claims.

Claims

1. An image classification method based on pruned neural network, characterized in that: include: Traversing the convolutional layers of the original image classification model, and constructing a Gaussian mixture model based on the weight data of the convolutional layers; Calculate the zero activation rate of the convolution kernel in each convolutional layer; Based on the Gaussian mixture model and the zero activation rate of the convolution kernel, a composite convolution association index of each convolution kernel is calculated, where the composite convolution association index is obtained by the zero activation rate, the similarity, and the overall distribution influence degree; And count the mean of the composite convolution correlation index of each convolution layer; Selecting the convolution layer with the highest mean value of the composite convolution correlation index as the target convolution layer, removing the convolution kernels whose composite convolution correlation index is greater than a preset threshold in the target convolution layer, and obtaining a pruned image classification model; The image data to be processed is input into the pruned image classification model to obtain image classification data.

2. The method according to claim 1, characterized in that Constructing a Gaussian mixture model based on the weight data of the convolutional layer, including: Extract the weight data of the convolutional layer; Constructing a Gaussian mixture model according to the weight data; The parameters of the Gaussian mixture model are solved based on the expectation maximization algorithm to obtain the mean, variance and probability of each Gaussian distribution obeying the weight of each Gaussian distribution.

3. The method according to claim 1, characterized in that: The composite convolution correlation index of each convolution kernel is calculated based on the Gaussian mixture model and the zero activation rate of the convolution kernel, including: Based on the Gaussian mixture model, calculating the similarity of each convolution kernel; Based on the Gaussian mixture model, the overall distribution influence of each convolution kernel is calculated; Based on the zero activation rate, the similarity and the sum of the overall distribution influence of each convolution kernel, the composite convolution association index of each convolution kernel is obtained.

4. The method according to claim 3, characterized in that Based on the Gaussian mixture model, the similarity of each convolution kernel is calculated, including: Based on the Gaussian mixture model, the probability that the weight data of the convolution kernel obeys each Gaussian distribution is obtained; Based on the probability that all weight data in the convolution kernel obey a certain Gaussian distribution and the ratio of the number of weights of the convolution kernel, the membership of the convolution kernel to the Gaussian distribution is obtained; Based on the membership of all Gaussian distributions of the convolution kernel, a membership set is obtained; Constructing a vector according to the membership set of the convolution kernels, and using the vector to calculate the cosine similarity between the convolution kernels; Based on the cosine similarity between the convolution kernels, the similarity of each convolution kernel is determined.

5. The method according to claim 4, characterized in that Based on the Gaussian mixture model, the overall distribution influence of each convolution kernel is calculated, including: Calculating the variance of the convolution kernel based on the membership set of the convolution kernel; The overall distribution influence of the convolution kernel is determined according to the variance.

6. The method according to claim 1, characterized in that After removing the convolution kernel whose composite convolution correlation index is greater than a preset threshold, the method further includes: Calculate the mean of the composite convolutional correlation index for each convolutional layer in the network; The convolution kernel of the convolution layer with the highest mean value of the composite convolution correlation index is selected for pruning until the mean value of the composite convolution correlation index of each convolution layer is lower than the preset threshold, or until the pruning rate is greater than the preset threshold, and then stop pruning.

7. The method according to claim 6, characterized in that After stopping pruning, it also includes: Retrain and fine-tune the parameters of the pruned image classification model.

8. An image classification device based on a pruned neural network, characterized in that: include: A construction module, used for traversing the convolutional layers of the original image classification model, and constructing a Gaussian mixture model based on the weight data of the convolutional layers; Zero activation rate calculation module, used to calculate the zero activation rate of the convolution kernel in each convolution layer; A correlation index calculation module, used to calculate the composite convolution correlation index of each convolution kernel based on the Gaussian mixture model and the zero activation rate of the convolution kernel, wherein the composite convolution correlation index is obtained by the zero activation rate, the similarity and the overall distribution influence degree; And count the mean of the composite convolution correlation index of each convolution layer; A pruning module is used to select the convolution layer with the highest mean value of the composite convolution correlation index as the target convolution layer, remove the convolution kernels whose composite convolution correlation index is greater than a preset threshold in the target convolution layer, and obtain a pruned image classification model; The image classification module is used to input the image data to be processed into the pruned image classification model to obtain image classification data.

9. An electronic device, characterized in that: It comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the image classification method based on pruning neural network as claimed in any one of claims 1 to 7 when executing the program instructions.

10. A computer-readable medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions are executed by a processor to implement an image classification method based on a pruned neural network as described in any one of claims 1 to 7.

Images