Image recognition network model pruning method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a kind of image recognition network model pruning method, device and equipment and storage medium, involve image processing technical field, wherein method includes: obtaining training dataset and image recognition network model, obtain the output feature map corresponding to each convolution kernel in image recognition network model, dimension reduction is carried out to output feature map, obtain the dimension reduction feature map corresponding to each convolution kernel, clustering is carried out to dimension reduction feature map and obtains clustering information, according to dimension reduction feature map, the pruning index corresponding to each convolution kernel is obtained, according to pruning index and clustering information, image recognition network model is pruned, and the light weight image recognition network model is obtained.This embodiment reduces the complexity and parameter amount of image recognition network model under the premise of not reducing accuracy index and model generalization ability, reduces the occupancy amount of operation resource, improves the running speed of model on resource limited device, expands the application range of image recognition network model.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image processing, in particular to a method and device for pruning an image recognition network model, a device and a storage medium. BACKGROUND

[0002] In recent years, with the development of computer technology and the continuous improvement of computing power and the explosive growth of data, artificial intelligence (AI) has rapidly penetrated into the medical field and shown great superiority in image recognition, big data mining and analysis. Medical companies use the feature extraction and data integration capabilities of artificial intelligence to develop automated white blood cell image recognition systems that have begun to enter medical laboratories, relieving the work pressure of laboratory personnel. Convolutional neural network (CNN) and other deep learning algorithms are a new image recognition algorithm that has emerged in recent years, which can automatically learn and integrate features from massive data, and its ability to detect image targets has been developed more and more deeply. In the field of medical image recognition, convolutional neural network technology can be used to study white blood cell morphology using image recognition, for example, based on multiple acute lymphoblastic leukemia L1, L2, L3 subtype bone marrow image and normal bone marrow image data sets, using a simple threshold segmentation method, a convolutional neural network model is trained to recognize immature lymphocytes and normal cells.

[0003] The convolutional neural network includes multiple convolution kernels to extract features from input data using multiple convolution kernels, and then obtain results using the extracted features. Generally, the more convolution kernels, the more features extracted, and the more accurate the final results. However, when the number of convolution kernels increases, the amount of calculation increases, and the computing resources consumed also increase, resulting in low computing efficiency. At the same time, due to the large amount of calculation and parameters, the deployment of high-precision deep learning algorithm models (such as convolutional neural network models) on resource-constrained devices, such as mobile devices / embedded devices, is limited to a certain extent. The edge platform has limited storage space and limited computing resources, resulting in low network running efficiency and poor economic applicability. SUMMARY

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] Embodiments of the present application provide a method and device for pruning an image recognition network model, a device and a storage medium, which can reduce the complexity and parameter quantity of the image recognition network model, reduce the occupation amount of operation resources and improve the running speed of the model on resource-constrained devices, and expand the application range of the image recognition network model.

[0006] In a first aspect, the embodiments of the present application provide a method for pruning an image recognition network model, comprising:

[0007] obtaining a training data set and an image recognition network model;

[0008] inputting the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model;

[0009] performing dimension reduction on the output feature map to obtain a dimension-reduced feature map corresponding to each convolution kernel;

[0010] performing clustering on the dimension-reduced feature map to obtain clustering information;

[0011] obtaining a pruning index corresponding to each convolution kernel according to the dimension-reduced feature map;

[0012] pruning the image recognition network model according to the pruning index and the clustering information to obtain a lightweight image recognition network model.

[0013] In an optional implementation, the training data set comprises training image samples and corresponding labels, and the inputting the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model comprises:

[0014] calculating a picture average value of the training image samples in the training data set;

[0015] inputting the picture average value into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model.

[0016] In an optional implementation, the output feature map is an output matrix, and the performing dimension reduction on the output feature map to obtain a dimension-reduced feature map corresponding to each convolution kernel comprises:

[0017] calculating eigenvalues and corresponding eigenvectors of the output matrix by using principal component analysis;

[0018] selecting the eigenvector with the largest eigenvalue as the dimension-reduced feature map corresponding to the output feature map.

[0019] In an optional implementation, the performing clustering on the dimension-reduced feature map to obtain clustering information comprises:

[0020] obtaining a density value of a point in each dimension-reduced feature map;

[0021] selecting at least one cluster center from the density value according to a local density value, a distance of a high local density point, and a corresponding preset distance threshold.

[0022] clustering the points in the reduced dimension feature map according to a clustering condition of the cluster centers, to obtain at least one cluster category.

[0023] In an optional implementation, the pruning index corresponding to each convolution kernel is obtained according to the reduced dimension feature map, including:

[0024] obtaining a channel scaling factor corresponding to each channel in each convolution kernel;

[0025] training the image recognition network model by using the model weight of each convolution kernel and the channel scaling factor, to adjust the channel scaling factor;

[0026] sorting the adjusted channel scaling factor, to obtain the pruning index corresponding to each convolution kernel.

[0027] In an optional implementation, the pruning the image recognition network model according to the pruning index and the clustering information, to obtain a lightweight image recognition network model, includes:

[0028] for the same cluster category, only retaining the model weight of the convolution kernel corresponding to the cluster center of the cluster category;

[0029] pruning the corresponding channel in the convolution kernel according to the pruning index, to obtain the lightweight image recognition network model.

[0030] In an optional implementation, the pruning the image recognition network model according to the pruning index and the clustering information, to obtain a lightweight image recognition network model, includes:

[0031] generating a base feature map according to the convolution kernel corresponding to the cluster center;

[0032] generating a similar feature map according to the base feature map;

[0033] adjusting the lightweight image recognition network model by using the base feature map and the similar feature map.

[0034] In an optional implementation, after obtaining the image recognition network model, the method further includes:

[0035] performing model optimization on the image recognition network model, the model optimization including at least one of width optimization, depth optimization, or resolution optimization.

[0036] In a second aspect, an embodiment of the present application provides a deep learning model pruning device, including:

[0037] The acquisition module is configured to acquire a training data set and an image recognition network model.

[0038] The output feature map acquisition module is configured to input the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model.

[0039] The dimension-reduced feature map acquisition module is configured to perform dimension reduction on the output feature map to obtain a dimension-reduced feature map corresponding to each convolution kernel.

[0040] The clustering module is configured to cluster the dimension-reduced feature map to obtain clustering information.

[0041] The pruning index calculation module is configured to obtain a pruning index corresponding to each convolution kernel according to the dimension-reduced feature map.

[0042] The pruning module is configured to prune the image recognition network model according to the pruning index and the clustering information to obtain a lightweight image recognition network model.

[0043] In a third aspect, a computer device includes a processor and a memory.

[0044] The memory is configured to store a program.

[0045] The processor is configured to execute the image recognition network model pruning method according to any one of the first aspect.

[0046] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium storing computer executable instructions for executing the image recognition network model pruning method according to any one of the first aspect.

[0047] Compared with the related art, the image recognition network model pruning method provided in the first aspect of the embodiments of the present application acquires a training data set and an image recognition network model, inputs the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model, performs dimension reduction on the output feature map to obtain a dimension-reduced feature map corresponding to each convolution kernel, clusters the dimension-reduced feature map to obtain clustering information, obtains a pruning index corresponding to each convolution kernel according to the dimension-reduced feature map, and prunes the image recognition network model according to the pruning index and the clustering information to obtain a lightweight image recognition network model. In this embodiment, the feature map is subjected to dimension reduction and clustering, the image recognition network model is pruned according to the clustering result and the pruning index to obtain a lightweight image recognition network model, the complexity and the parameter amount of the image recognition network model are reduced on the premise of not reducing the accuracy index and the model generalization capability, the occupation amount of the operation resource is reduced, the running speed of the model on a resource-limited device is improved, and the application range of the image recognition network model is expanded.

[0048] It can be understood that the beneficial effects of the second aspect to the fourth aspect compared with the related art are the same as the beneficial effects of the first aspect compared with the related art, which can be seen from the related description in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or related description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0050] Figure 1 is a flowchart of the image recognition network model pruning method provided by an embodiment of the present application;

[0051] Figure 2 is another flowchart of the image recognition network model pruning method provided by an embodiment of the present application;

[0052] Figure 3 is another flowchart of the image recognition network model pruning method provided by an embodiment of the present application;

[0053] Figure 4 is another flowchart of the image recognition network model pruning method provided by an embodiment of the present application;

[0054] Figure 5 is another flowchart of the image recognition network model pruning method provided by an embodiment of the present application;

[0055] Figure 6 is a flowchart of the image recognition network model pruning method provided by an embodiment of the present application;

[0056] Figure 7 is a structural schematic diagram of the image recognition network model pruning device provided by an embodiment of the present application;

[0057] Figure 8 is a hardware structural schematic diagram of the electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0059] It is noted that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be performed in an order different from that in the flowcharts. The terms "first", "second", and the like in the description and in the claims, as well as above-described appended drawings, are used for distinguishing between similar objects and not necessarily for describing a specific sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of use in either order.

[0060] It is also to be understood that the phraseology "one embodiment" or "some embodiments" used in the specification and throughout this description is not used to refer to one or several embodiments in particular, but rather to one or more embodiments, unless otherwise indicated. The terms "including", "comprising", "having" and variations thereof herein are meant to encompass the presence of stated features, steps, components, or integers, but not the exclusion of others thereof. The terms "including", "comprising", "having" and variations thereof are meant not to be limiting.

[0061] Leukemia is a common hematological malignancy with high morbidity and mortality. In China, the incidence of leukemia is among the highest among all tumors, and leukemia is also the tumor with the highest incidence and mortality in children. Rapid, safe and accurate diagnosis of leukemia is an effective means to reduce morbidity and mortality. Morphological assessment and reliable identification of malignant white blood cells are key steps in diagnosing leukemia. Previously, professional inspectors relied on manual microscopy to judge cells in peripheral blood and bone marrow blood sample blood smears, and finally draw a classification conclusion.

[0062] In recent years, with the development of computer technology and the continuous improvement of computing power and the explosive growth of data, artificial intelligence (AI) has rapidly penetrated into the medical field and shown great superiority in image recognition, big data mining and analysis. Medical companies can now use the feature extraction and data integration capabilities of artificial intelligence to develop automated white blood cell image recognition systems that have begun to enter medical laboratories, relieving the work pressure of laboratory personnel. Convolutional neural network (CNN) and other deep learning algorithms are a new image recognition algorithm that has emerged in recent years, which can automatically learn and integrate features from massive data, and its ability to detect image targets has been increasingly developed. In the field of medical image recognition, convolutional neural network technology can be used to conduct research on leukemia cell morphology using image recognition. For example, based on multiple acute lymphoblastic leukemia L1, L2, L3 subtype bone marrow image and normal bone marrow image data sets, a simple threshold segmentation method is used to train a convolutional neural network model to recognize immature lymphocytes and normal cells.

[0063] The convolutional neural network includes multiple convolution kernels to extract features from input data using multiple convolution kernels, and then obtain results using the extracted features. Generally, the more convolution kernels, the more features extracted, and the more accurate the final results obtained. However, when the number of convolution kernels increases, the amount of calculation increases, and the computing resources consumed also increase, resulting in low computing efficiency. With the development of the times, people pay more attention to the practical application performance of neural networks. Due to the large amount of calculation and the large number of parameters, to some extent, the deployment of high-precision deep learning algorithm models (such as convolutional neural network models) on resource-constrained devices, such as mobile devices / embedded devices, etc. The edge platform has limited storage space and limited computing resources, resulting in low network running efficiency and poor economic applicability. Therefore, it is necessary to study a lightweight method based on a deep learning model. In related technologies, there are a lot of redundancies in the parameters of many studies, especially the small size of the data set trained by the existing technology, which is easy to cause overfitting, resulting in insufficient generalization ability of the model and low precision in practical application. In addition, although some studies have reduced the number of parameters to some extent, the storage space of the model itself is also very large.

[0064] In related technologies, channel pruning is one of the methods to realize model lightweight. How to measure the importance of weights is the core problem of pruning algorithms. The pruning method in the past often involves a method to measure the importance of weights itself to realize model compression, but it is difficult to measure the influence of channels on the results and determine the pruning rate.

[0065] Therefore, compared with related technologies, the method for pruning the image recognition network model provided in the embodiments of the present application obtains a training data set and an image recognition network model, inputs the training data set into the image recognition network model, obtains an output feature map corresponding to each convolution kernel in the image recognition network model, performs dimension reduction on the output feature map to obtain a dimension-reduced feature map corresponding to each convolution kernel, performs clustering on the dimension-reduced feature map to obtain clustering information, obtains a pruning index corresponding to each convolution kernel according to the dimension-reduced feature map, prunes the image recognition network model according to the pruning index and the clustering information, and obtains a lightweight image recognition network model. In the embodiments, the dimension-reduced feature map is clustered, the image recognition network model is pruned according to the clustering result and the pruning index to obtain the lightweight image recognition network model, the complexity and the parameter amount of the image recognition network model are reduced on the premise that the accuracy index and the model generalization capability are not reduced, the occupation amount of the operation resource is reduced, the running speed of the model on a resource-limited device is improved, and the application range of the image recognition network model is expanded.

[0066] The embodiments of the present application are further described below with reference to the drawings.

[0067] The method for pruning the image recognition network model provided in the embodiments of the present application relates to the technical field of artificial intelligence, in particular to the technical field of data mining. The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology and application system for using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use the knowledge to obtain the best results. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technologies mainly include computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0068] The image recognition network model pruning method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the software can be an application that implements the image recognition network model pruning method, etc., but is not limited to the above forms.

[0069] The embodiments of this application can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0070] The application scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will know that with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0071] Based on the above application scenarios, various embodiments of the image recognition network model pruning method of this application are proposed.

[0072] like Figure 1 As shown, Figure 1 This is a flowchart of an image recognition network model pruning method provided in one embodiment of this application, including but not limited to steps S110 and S160.

[0073] Step S110: Obtain the training dataset and image recognition network model.

[0074] In an embodiment, the image recognition network model can be a convolutional neural network model for performing white blood cell classification recognition, and the image recognition network model is trained using the convolutional neural network model to recognize immature lymphocytes and normal cells for subsequent white blood cell morphology analysis.

[0075] In an embodiment, the image recognition network model can be an EfficientNet network structure model, for example, an EfficientNet-B7 model, and the image recognition network model has a total of 813 layers, each layer is combined by 5 MBConv modules, the MBConv module is an attention mechanism module, which is mainly used to make the MBConv module network pay more attention to important parts, and can be used at the start of a sub-block. The start of the first sub-block of all main blocks in the image recognition network model, except for the first one, has a skip connection between all sub-blocks, and each sub-block is connected and combined with the previous sub-block in a skip connection manner to improve the convergence speed of the model.

[0076] In this embodiment, the MBConv module first performs 1*1 pointwise convolution on the input to change the dimension of the output channel, then performs k*k (k can be set according to experience) depth separable convolution, and then uses 1*1 pointwise convolution to restore the channel dimension and connects to the previous convolution layer in a skip connection manner. At the same time, the calculation stage of the feature map with the same size is taken as the same stage, and each down-sampling stage is taken as the next stage, Cross Stage Partial Network components are added at the beginning and end of the same stage to reduce the amount of calculation while maintaining or improving the accuracy, realize feature fusion between different feature maps, further strengthen information flow, and improve the network performance. And add channel attention mechanism and spatial attention mechanism at the end of each stage to help the image recognition network model assign different weights to each part of the input, so as to extract more critical information and further improve the robustness of the image recognition network model.

[0077] In an embodiment, since the above-mentioned image recognition network model is a fast and high-precision model, a depth, width, and resolution (i.e., input picture resolution) common adjustment technique is used, and therefore the model optimization of the above-mentioned image recognition network model is also included, wherein the model optimization includes at least one of width optimization, depth optimization, or resolution optimization.

[0078] In this embodiment, the model optimization is to take the above image recognition network model as a benchmark model, and to make deep adjustment on the basis of the benchmark model, i.e. to adjust the channel number of the picture; or to make deep adjustment on the basis of the benchmark model, i.e. to adjust the number of layers of the network; or to make adjustment on the size of the input picture on the basis of the benchmark model (i.e. to adjust the resolution); or to make adjustment on the depth, width and size of the picture on the basis of the benchmark model, etc.

[0079] In an embodiment, the above model optimization process includes the following steps:

[0080] 1) Introduce network parameter Φ and set the initial value: Φ = 1, and then find the parameter that satisfies the following formula through grid search:

[0081] Depth: d = α Φ ,;

[0082] Width: ω = β Φ ;

[0083] Resolution: r = γ Φ ;

[0084] Wherein, α·β 2 ·γ 2 ≈ 2, and α≥1, β≥1, γ≥1;

[0085] Find the optimal α, β, γ, so that accuracy (N (d, ω, r)) is maximum, accuracy represents the classification accuracy of the image recognition network model, i.e. the highest accuracy of α, β, γ under the condition of parameters (d, ω, r).

[0086] 2) Under the condition of the above obtained α, β, γ parameters, set different network parameters Φ, train the image recognition network model (such as EfficientNet-B7 network model), so that the parameter quantity and the calculation quantity are minimum.

[0087] The training strategy used in the above step 2) can be:

[0088] (1) Multi-scale training strategy: i.e. to define several fixed scales in advance, and to randomly select a scale for training in each epoch to enhance the robustness of the image recognition network model;

[0089] (2) Random dropout training strategy: i.e. to randomly make some nodes in the image recognition network model not work (i.e. to set the output of the node to 0) during the training of the image recognition network model, and the other processes remain unchanged, to prevent the problem of overfitting during training;

[0090] (3) Class balanced loss training strategy: i.e. to prevent small sample feature loss by balancing the weights of different cell classes in the training dataset.

[0091] (4) Crop Augmentation and rotation training strategy: i.e. to improve the robustness of the network to different cell sizes and angles by randomly cropping a predefined size region from the picture and rotating it at a random angle during training.

[0092] The optimized image recognition network model is obtained through the above training and testing.

[0093] In an embodiment, for example, taking the white blood cells of acute myeloid leukemia patients as the recognition object, the dataset can use the microscope picture dataset of acute myeloid leukemia white blood cells that has been correctly classified and labeled. The dataset can be a newly established white blood cell image library or a publicly available white blood cell image library. The white blood cell microscope pictures can come from artificial microscope examination pictures of peripheral blood smears, artificial microscope examination pictures of bone marrow blood smears, or artificial microscope examination pictures of mixed blood smears from both. In addition, the blood samples for generating the blood smears include normal blood cell samples and abnormal blood cell samples.

[0094] In this embodiment, image preprocessing techniques can also be used to adjust the image size in the training dataset. For example, data augmentation techniques such as flipping, rotating, or color normalization can be used to enhance the initial dataset. In addition, the processed white blood cell microscope picture dataset can be randomly divided into a white blood cell classification training dataset and a test dataset in a ratio of 8:2. The training dataset is used for parameter training of the image recognition network model (such as a convolutional neural network model), and the test dataset is used to verify the accuracy of the image recognition network model and update the parameter weights.

[0095] It can be understood that the above-mentioned leukemia cells can be replaced by other cells, including but not limited to normal white blood cells, abnormal lymphocytes, neutrophils, pathogens, etc. These cells can come from peripheral blood, bone marrow, cerebrospinal fluid, alveolar lavage fluid, urine, etc. Further, replacing other blood cells can assist in the diagnosis of different blood diseases such as other leukemias and their subtypes, lymphomas, etc.; replacing cells from cerebrospinal fluid, alveolar lavage fluid, and urine can assist in the diagnosis of brain, lung, eye, etc. tumors or infectious diseases, etc. The training dataset is not specifically limited here, and the image recognition model of the present embodiment can be applied to various fields involving image recognition.

[0096] Step S120, input the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model.

[0097] In an embodiment, the training data set includes training image samples and corresponding labels. In this embodiment, a picture average value of each training image sample in the training data set is calculated and input into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model. Specifically, the output feature map is an output matrix of n rows and m columns.

[0098] Step S130, dimension reduction is performed on the output feature map to obtain a dimension-reduced feature map corresponding to each convolution kernel.

[0099] In an embodiment, step S130 includes but is not limited to the following steps:

[0100] Step S131, the eigenvalue and the corresponding eigenvector of the output matrix are calculated by using the principal component analysis method.

[0101] Step S132, the eigenvector with the largest eigenvalue is selected as the dimension-reduced feature map corresponding to the output feature map.

[0102] In an embodiment, in order to analyze the distribution of the convolution kernel in the network, the layer with more redundancy needs to be selected as the layer to be pruned by the weight value of the BN layer. In this embodiment, for each convolution layer to be pruned, the output feature map of the layer is obtained through forward inference, and then the principal component analysis (PCA) method is used to reduce the data from two dimensions to one dimension through orthogonal transformation to obtain the eigenvector of the output feature map.

[0103] The principal component analysis method is a multivariate statistical method and one of commonly used dimension reduction methods. It converts a group of variables that may have correlations into a group of linearly uncorrelated variables through orthogonal transformation, and the converted variables are called principal components. Two methods can be used for principal component analysis, namely eigenvalue decomposition or singular value decomposition.

[0104] In this embodiment, the covariance matrix of each output matrix is calculated by using the principal component analysis method, and the corresponding eigenvalue and eigenvector are obtained according to the covariance matrix. Then, the eigenvector corresponding to the largest eigenvalue is taken as the dimension-reduced feature map corresponding to the output feature map.

[0105] Step S140, clustering is performed on the dimension-reduced feature map to obtain clustering information.

[0106] In an embodiment, an unsupervised clustering method is adopted. Clustering is a process of partitioning a set of data objects into clusters of similar objects. Objects in the same cluster (or class) have high similarity, while objects in different clusters have high dissimilarity. According to the scale of clustering, clustering methods can be divided into three types: distance-based clustering algorithm, density-based clustering method, and connectivity-based clustering algorithm. Among them, the distance-based clustering algorithm uses various distances to measure the similarity between data objects. The density-based clustering algorithm is mainly based on a suitable density function. The connectivity-based clustering algorithm is usually based on a graph or hypergraph model, and highly connected objects are clustered into a class.

[0107] In this embodiment, a local density-based clustering method is adopted to cluster the plurality of reduced dimension feature maps obtained above. The local density-based clustering method can cluster arbitrary shape data and avoids the defects of the k-means clustering method which requires pre-specification of the class and the DBSCAN clustering method which requires pre-specification of the density threshold, thereby obtaining the clustering results of each reduced dimension feature map.

[0108] In an embodiment, the density-based clustering method for clustering the reduced dimension feature maps to obtain the clustering information includes but is not limited to the following steps: Figure 2

[0109] Step S141, obtaining the density value of each point in each reduced dimension feature map.

[0110] In an embodiment, the Wasserstein distance is used as the distance measure between the reduced dimension feature maps, and the specific clustering idea is: it is assumed that the center of the class cluster is surrounded by some points with low local density, and the distance (i.e. the Wasserstein distance) between these points and other points with high local density is relatively large.

[0111] First, two values are set: the local density value ρ i and the distance δ i of the high local density point.

[0112] The above two values only depend on the distance d ij between two points i and j, and the distance satisfies the triangle inequality:

[0113] ρ i =∑ j χ(d ij -d c )

[0114] where d c represents the optimal cut-off distance, which is a hyperparameter and can be obtained empirically. Therefore, ρ i is equivalent to the distance from point i being less than d c .The number of points, since this algorithm only works for ρ i Sensitive to the relative value of d, therefore sensitive to d c The choice is more robust.

[0115] δ i The minimum distance between point i and other points with high local density, i.e., the shortest distance between all points with higher density than point i, is expressed as:

[0116]

[0117] For the point with the highest density, set δ i =min j (d ij Only those points with the highest local or global density will have a density much greater than the normal distance between adjacent points; therefore, cluster centers are considered to be δ. i The point with the largest abnormal value.

[0118] Step S142, based on the local density value ρ i δ, the distance between high local density points i And the corresponding preset distance threshold, and select at least one cluster center from the density values.

[0119] Step S143: Cluster the points in the dimensionality-reduced feature map according to the clustering conditions of the cluster centers to obtain at least one cluster category.

[0120] In one embodiment, the density values ​​of all points are arranged from high to low, and the distance is determined by a given preset distance threshold, δ. min and ρ min Filter out those that simultaneously satisfy: 1) ρ i >ρ min ;2)δ i >δ min The points that meet the conditions are used as the distances from the center point, and thus serve as the cluster centers.

[0121] In this embodiment, the clustering process can be described as: having a relatively large local density value ρ i And a large δ i The point is considered the center of the cluster, while the local density ρ i Smaller but δ i Larger points are considered outliers. The point with the highest local density and the largest distance to other points with higher local density is selected as the cluster center. After determining the cluster centers, the class labels of the remaining points are assigned according to the principle that the class label of the current point is equal to the label of the nearest point with a higher density. This determines the number of clusters. All other points belong to the cluster represented by their nearest cluster center, and the remaining points are assigned to the same cluster as their nearest neighbor with higher density.

[0122] In the clustering process of the embodiment, the reliability of dividing each point to a certain cluster needs to be determined, and therefore, in the embodiment, a clustering boundary is defined for each cluster, and the maximum density value of all points in a cluster to all points in other clusters with a distance less than or equal to p is calculated to obtain the clustering boundary. The points with a distance less than d to other clusters can be selected according to the clustering boundary, and then the point with the maximum local density of the clustering boundary is found for each cluster, and the local density is denoted as d, and all points with a local density greater than d are considered as a part of the cluster core, that is, the reliability of dividing the point to the cluster is great, and the remaining points are considered as abnormal points or noise of the cluster. In the embodiment, the clustering process of the non-clustering center points is separated into a single process, so that the selection of the clustering center and the classification of the non-clustering points are separated, the clustering accuracy is increased, and the clustering of pictures and non-spherical point sets is applicable. i c min min In the embodiment, the clustering process of the non-clustering center points is separated into a single process, so that the selection of the clustering center and the classification of the non-clustering points are separated, the clustering accuracy is increased, and the clustering of pictures and non-spherical point sets is applicable.

[0123] In step S150, a pruning index corresponding to each convolution kernel is obtained according to the dimension-reduced feature map.

[0124] In an embodiment, referring to Figure 3 , step S150 includes but is not limited to the following steps:

[0125] In step S151, a channel scaling factor corresponding to each channel in each convolution kernel is obtained.

[0126] In an embodiment, a channel scaling factor is introduced for each channel in the convolution kernel, which is used to multiply the output of the corresponding channel. The scaling factor can be selected according to an empirical value or actual demand.

[0127] In step S152, the image recognition network model is trained by using the model weight of each convolution kernel and the channel scaling factor, so as to adjust the channel scaling factor.

[0128] In an embodiment, the image recognition network model is jointly trained by using the model weight of each convolution kernel and the channel scaling factor, and the channel scaling factor is adjusted in the training process.

[0129] In step S153, the adjusted channel scaling factor is sorted to obtain a pruning index corresponding to each convolution kernel.

[0130] In an embodiment, the absolute values of the adjusted channel scaling factors are sorted from large to small or from small to large, and the channel corresponding to the smaller channel scaling factor is taken as the pruning index corresponding to each convolution kernel, that is, the pruning channel can be selected according to the pruning index.

[0131] ​​​Step S160, pruning the image recognition network model according to the pruning index and the clustering information to obtain a lightweight image recognition network model.

[0132] In an embodiment, referring to Figure 4 , step S160 includes but is not limited to the following steps:

[0133] Step S161, for the same clustering category, only retaining the model weight of the convolution kernel corresponding to the clustering center of the clustering category.

[0134] Step S162, pruning the corresponding channel in the convolution kernel according to the pruning index to obtain a lightweight image recognition network model.

[0135] In an embodiment, for the same clustering category, only the model weight of the convolution kernel corresponding to the clustering center of the clustering category is retained, and the model weight of the convolution kernel generating other reduced dimension feature maps in the clustering category is pruned, that is, only the model weight of the convolution kernel corresponding to the clustering center is retained in the same clustering category.

[0136] In addition, the channel corresponding to the non-clustering center is pruned, that is, the channel corresponding to the smaller channel scaling factor is pruned according to the pruning index, and the number of pruned channels is selected according to the system lightweight demand, so as to obtain a lightweight image recognition network model.

[0137] For the lightweight image recognition network model obtained above, in order to avoid over-pruning, which leads to the decline of the inference ability of the image recognition network model, that is, the decline of the detection accuracy, it is necessary to adjust the lightweight image recognition network model.

[0138] In an embodiment, referring to Figure 5 , the step of adjusting the lightweight image recognition network model includes but is not limited to:

[0139] Step S510, generating a basic feature map according to the convolution kernel corresponding to the clustering center.

[0140] Step S520, generating a similar feature map according to the basic feature map.

[0141] Step S530, adjusting the lightweight image recognition network model by using the basic feature map and the similar feature map.

[0142] In an embodiment, the reserved cluster center corresponding convolution kernel is used to generate the base feature map of each layer. Since the feature representations of the relevant feature maps of the same cluster category have great similarity and redundancy, and a linear transformation method can be used to obtain a feature map similar to another feature map through a certain feature map, linear operation and nonlinear operation are used in this embodiment to generate similar feature maps from the base feature map as a supplement. The similar feature maps can fully reveal the potential inherent feature information of the base feature map, so as to reduce the model parameter amount while ensuring the inference ability of the model as much as possible.

[0143] In an embodiment, two layers of 1*1 convolution in series with very small parameter amount are used to simulate the linear operation and nonlinear operation described above. The linear calculation simply processes the gray value of each pixel in the base feature map (for example, multiplying or dividing a weight value), and then performs summation. The nonlinear calculation is to find the maximum value, minimum value, median value, mean value, etc. in a certain range (for example, 3 pixels*3 pixels) around each pixel in the base feature map.

[0144] Specifically, for example, a Relu() function is connected after the first convolution layer of the convolution kernel to achieve the purpose of nonlinear operation, and the second convolution layer is not connected with the Relu() function to achieve the purpose of linear operation. Since linear operation and nonlinear operation are added, the weights of the two convolution layers in the convolution kernel need to be adjusted through training to achieve the purpose of restoring the precision of the lightweight image recognition network model. In an embodiment, the lightweight image recognition network model is retrained after adjusting the structure of the lightweight image recognition network model, and then performance testing is performed. The recognition accuracy of the lightweight image recognition network model can reach 99.27%, and the storage space occupied by the lightweight image recognition network model is only 16M.

[0145] The image recognition network model pruning method of the embodiments of the present application can solve the problems of low network running efficiency and poor economic applicability caused by the complex structure, large amount of operation and large amount of parameter redundancy of the existing image recognition network model, greatly reduce the model parameter amount and calculation amount under the condition that the accuracy index is similar to the existing classification network, and ensure that the lightweight image recognition network model has high accuracy, occupies small storage space and runs fast, which is conducive to promoting the landing of deep learning in the medical image recognition field.

[0146] In an embodiment, reference is made to Figure 6 The flowchart of the image recognition network model pruning method in an embodiment of the present application is shown in FIG. 1.

[0147] Figure 6The training data set is first acquired, and then the training sample is input into an image recognition network model, the image recognition network model comprising a plurality of convolution layers (for example, n in the figure), each convolution layer comprising a plurality of convolution kernels. The training data set is input into the optimized image recognition network model, and an output feature map corresponding to each convolution kernel in the image recognition network model is obtained. The output feature map is dimensionally reduced to obtain a dimensionally reduced feature map corresponding to each convolution kernel, wherein the output feature map is represented in the form of an output matrix. Then, eigenvalues and corresponding eigenvectors of the output matrix are calculated, and a dimensionally reduced feature map corresponding to the output feature map is obtained from the eigenvalues and the corresponding eigenvectors. The dimensionally reduced feature map is clustered to obtain clustering information, wherein the clustering information includes clustering categories, clustering centers (the clustering centers are represented by diagonal lines in the figure), clustering boundaries, and the like. On the basis of pruning the channels corresponding to the convolution kernels, only the model weight of the convolution kernel corresponding to the clustering center is retained for the same clustering category, and the other convolution kernels are pruned, thereby obtaining a lightweight image recognition network model.

[0148] Further, with reference to Figure 6 In order to avoid excessive pruning and cause the inference ability of the image recognition network model to decrease, the lightweight image recognition network model is adjusted through linear transformation and nonlinear transformation. A base feature map is generated according to the convolution kernel corresponding to the clustering center, a similar feature map is generated according to the base feature map, and the lightweight image recognition network model is adjusted by using the base feature map and the similar feature map.

[0149] Compared with related technologies, the method provided by the embodiment of the present application acquires a training data set and an image recognition network model, inputs the training data set into the image recognition network model, obtains an output feature map corresponding to each convolution kernel in the image recognition network model, dimensionally reduces the output feature map to obtain a dimensionally reduced feature map corresponding to each convolution kernel, clusters the dimensionally reduced feature map to obtain clustering information, obtains a pruning index corresponding to each convolution kernel according to the dimensionally reduced feature map, prunes the image recognition network model according to the pruning index and the clustering information, and obtains a lightweight image recognition network model. The embodiment dimensionally reduces and clusters the feature map, prunes the image recognition network model according to the clustering result and the pruning index to obtain the lightweight image recognition network model, adopts a method combining feature image similarity and unsupervised clustering to realize adaptive model pruning, measures the weight importance of the convolution kernel, and automatically calculates the pruning index of each convolution layer, thereby avoiding negative consequences caused by manually setting a threshold. The method can reduce the complexity and the parameter quantity of the image recognition network model without reducing the accuracy index and the model generalization ability, reduce the occupied amount of operation resources, improve the running speed of the model on a resource-limited device, and expand the application range of the image recognition network model.

[0150] In addition, an embodiment of an embodiment of the present application further provides a deep learning model pruning device, with reference toFigure 7 The device comprises:

[0151] The acquisition module 710 is configured to acquire a training data set and an image recognition network model.

[0152] The output feature map acquisition module 720 is configured to input the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model.

[0153] The dimension-reduced feature map acquisition module 730 is configured to perform dimension reduction on the output feature map to obtain a dimension-reduced feature map corresponding to each convolution kernel.

[0154] The clustering module 740 is configured to cluster the dimension-reduced feature map to obtain clustering information.

[0155] The pruning index calculation module 750 is configured to obtain a pruning index corresponding to each convolution kernel according to the dimension-reduced feature map.

[0156] The pruning module 760 is configured to prune the image recognition network model according to the pruning index and the clustering information to obtain a lightweight image recognition network model.

[0157] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present embodiment scheme.

[0158] It should be noted that the deep learning model pruning device in the present embodiment can execute the image recognition network model pruning method in the embodiment shown in Figure 1 . That is, the deep learning model pruning device in the present embodiment and the image recognition network model pruning method in the embodiment shown in Figure 1 all belong to the same inventive concept, so these embodiments have the same implementation principles and technical effects, which will not be described in detail here.

[0159] In addition, one embodiment of the present embodiment also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor.

[0160] The processor and the memory can be connected through a bus or other means.

[0161] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0162] The non-transient software program and instructions required to implement the image recognition network model pruning method of the above embodiments are stored in memory. When executed by a processor, the image recognition network model pruning method of the above embodiments is executed, for example, the method described above is executed. Figure 1 Method steps S110 to S160 in the text Figure 2 Method steps S141 to S143, Figure 3 The method steps S151 to S153, etc.

[0163] Reference Figure 8 , Figure 8 This application illustrates the hardware structure of an electronic device according to an embodiment of the present application. The electronic device includes:

[0164] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0165] The memory 802 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and called and executed by the processor 801 using the blood cell information prediction method of the embodiments of this disclosure.

[0166] The 803 input / output interface is used to implement information input and output.

[0167] The communication interface 804 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth and the like).

[0168] The bus 805 is configured to transmit information between various components (for example, the processor 801, the memory 802, the input / output interface 803 and the communication interface 804) of the device.

[0169] The processor 801, the memory 802, the input / output interface 803 and the communication interface 804 are connected to each other through the bus 805 to realize the communication connection between the device.

[0170] In addition, one embodiment of the embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions. The computer executable instructions are executed by a processor or a controller, for example, are executed by a processor in the above computer device embodiment, so that the above processor executes the image recognition network model pruning method in the above embodiment, for example, executes the method steps S110 to S160 in the above method, Figure 1 the method steps S141 to S143 in the above method, Figure 2 the method steps S151 to S153 in the above method, Figure 3 and the like.

[0171] For another example, the computer executable instructions are executed by a processor in the above computer device embodiment, so that the above processor executes the image recognition network model pruning method in the above embodiment, for example, executes the method steps S110 to S160 in the above method, Figure 1 the method steps S141 to S143 in the above method, Figure 2 the method steps S151 to S153 in the above method, Figure 3 and the like.

[0172] As will be appreciated by one of ordinary skill in the art, all or some steps, systems of the above-disclosed methods can be implemented as software, firmware, hardware, or any suitable combination thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application- specific integrated circuit. Such software can be distributed on computer readable media, which can comprise computer storage media (or non-transitory media), and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. Further, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media.

[0173] The above is the specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above implementation, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the embodiments of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A method for pruning an image recognition network model, characterized in that, The method comprises the following steps: obtaining a training data set and an image recognition network model; inputting the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model, wherein the output feature map is an output matrix; calculating eigenvalues and corresponding eigenvectors of the output matrix by using principal component analysis, and selecting the eigenvector with the largest eigenvalue as a reduced dimension feature map corresponding to the output feature map of the convolution kernel; obtaining a density value of each point in each reduced dimension feature map, selecting at least one cluster center from the density value according to a local density value, a distance of a high local density point and a corresponding preset distance threshold, clustering the points in the reduced dimension feature map according to a clustering condition of the cluster center to obtain at least one clustering category; obtaining a channel scaling factor corresponding to each channel in each convolution kernel, training the image recognition network model by using a model weight of each convolution kernel and the channel scaling factor to adjust the channel scaling factor, sorting the adjusted channel scaling factor to obtain a pruning index corresponding to each convolution kernel; for the same clustering category, only retaining the model weight of the convolution kernel corresponding to the cluster center of the clustering category, and pruning the corresponding channel in the convolution kernel according to the pruning index to obtain a lightweight image recognition network model.

2. The method of claim 1, wherein, The training data set comprises training image samples and corresponding labels, and the inputting the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model comprises the following steps: calculating a picture average value of the training image samples in the training data set; inputting the picture average value into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model.

3. The method of claim 1, wherein, For the same clustering category, only retaining the model weight of the convolution kernel corresponding to the cluster center of the clustering category to obtain the lightweight image recognition network model further comprises the following steps: generating a basic feature map according to the convolution kernel corresponding to the cluster center; generating a similar feature map according to the basic feature map; adjusting the lightweight image recognition network model by using the basic feature map and the similar feature map.

4. The method of claim 1 to 3, wherein, After obtaining the image recognition network model, the method further comprises the following steps: performing model optimization on the image recognition network model, wherein the model optimization comprises at least one of width optimization, depth optimization or resolution optimization.

5. An image recognition network model pruning device, characterized in that, The method comprises the following steps: an obtaining module, configured to obtain a training data set and an image recognition network model; an output feature map obtaining module, configured to input the training data set into the image recognition network model to obtain an output feature map corresponding to each convolution kernel in the image recognition network model, wherein the output feature map is an output matrix; a reduced dimension feature map obtaining module, configured to calculate eigenvalues and corresponding eigenvectors of the output matrix by using principal component analysis, and select the eigenvector with the largest eigenvalue as a reduced dimension feature map corresponding to the output feature map of the convolution kernel; The clustering module is configured to obtain a density value of each point in the dimension-reduced feature map, select at least one cluster center from the density value according to the local density value, the distance of the high local density point, and a corresponding preset distance threshold, cluster the points in the dimension-reduced feature map according to a clustering condition of the cluster center, and obtain at least one cluster category; The pruning index calculation module is configured to obtain a channel scaling factor corresponding to each channel in each convolution kernel, train the image recognition network model by using a model weight of each convolution kernel and the channel scaling factor, adjust the channel scaling factor, sort the adjusted channel scaling factor, and obtain a pruning index corresponding to each convolution kernel; The pruning module is configured to retain only the model weight of the convolution kernel corresponding to the cluster center of the same cluster category, and prune the corresponding channel in the convolution kernel according to the pruning index, to obtain a lightweight image recognition network model.

6. A computer device, comprising: The processor and the memory are included. The memory is configured to store a program. The processor is configured to execute the image recognition network model pruning method according to any one of claims 1 to 4.

7. A computer readable storage medium storing computer executable instructions for executing the image recognition network model pruning method according to any one of claims 1 to 4.

Images