Methods and neural network devices for training neural networks for image recognition

Abstract

A method and a neural network device for training a neural network for image recognition are provided. The method sets weight thresholds based on the weight distribution of multiple layers included in the neural network, predicts changes in the inference accuracy of the neural network by pruning each layer based on the weight thresholds, determines the current target layer among the multiple layers included in the neural network to be pruned using the weight thresholds, and prunes the determined current target layer.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0113527, filed on September 16, 2019, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to image recognition, and more specifically, to neural network methods and devices for image recognition. Background Technology

[0003] Neural networks are computing systems implemented using processors that reference computational architectures.

[0004] Neural network devices for image recognition perform numerous complex operations on input image data to perform image recognition. As the number of input images and training operations increases, the connectivity of the neural network architecture becomes more complex. Accuracy on previously trained images increases, but overfitting problems arise: the reliability of predictions for new images decreases relative to previous training images, rather than accuracy increases. Furthermore, the increased complexity of neural networks leads to an excessive increase in memory allocation, resulting in inefficient performance for miniaturization and commercialization of related devices. Summary of the Invention

[0005] This summary is provided to introduce, in a simplified form, the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] In one general aspect, a method for training a neural network for image recognition includes: acquiring a first image set; pre-training the neural network based on the first image set; obtaining weights for multiple layers of the pre-trained neural network; setting a weight threshold for pruning the neural network to a target pruning rate based on a determined weight distribution; pruning the multiple layers of the neural network based on the weight threshold; predicting changes in inference accuracy of the pruned multiple layers of the neural network based on a pruned dataset input as a second image set; determining a current target layer among the multiple layers of the neural network to be pruned based on the predicted changes in inference accuracy; and generating a pruned neural network by pruning the determined current target layer.

[0007] The trimmed dataset can be a predetermined number of images randomly extracted from each class included in a given image set.

[0008] The method may further include: determining the weight distribution of the plurality of layers of the neural network.

[0009] The current target layer can be identified as the layer, excluding previously pruned layers, that is predicted to have the lowest sensitivity to changes in the inference accuracy of the prediction.

[0010] The step of predicting changes in inference accuracy may include: calculating the sensitivity of each of the plurality of layers based on the difference between the inference accuracy before pruning each of the plurality of layers and the inference accuracy after pruning each of the plurality of layers.

[0011] The layer predicted to have the lowest sensitivity corresponds to the layer predicted to have the least impact on the reduction of inference accuracy of the neural network.

[0012] The steps for setting the weight threshold may include: when the determined weight distribution corresponds to a standard normal distribution, setting the weight value corresponding to the target pruning rate as the weight threshold.

[0013] The steps for pruning the current target layer may include: adjusting the pruning rate of the weights of the current target layer by updating the weight threshold until the inference accuracy of the neural network based on the pruned dataset decreases to the threshold accuracy, thereby pruning the current target layer.

[0014] The step of updating the weight threshold may include: increasing the current weight threshold when the inference accuracy of the neural network, which includes weights pruned to the current weight threshold, is not reduced to the threshold accuracy.

[0015] The determination and pruning of the current target layer can be repeated until the target pruning rate is reached, or until the pruning of all layers of the neural network or a predetermined number of layers is completed.

[0016] The method may further include providing a trimmed dataset, wherein the trimmed dataset may include one of the following: an image set generated by randomly extracting a predetermined number of images for each class included in a given image set, and an image set generated by selecting valid classes from the trimmed dataset and randomly extracting a predetermined number of images for each selected valid class.

[0017] The steps of providing a pruned dataset may include: randomly extracting samples of a predetermined number of images from each class included in a given image set; determining a label corresponding to each sample in the randomly extracted samples by performing inference on the randomly extracted samples using a neural network; identifying the classes among the classes included in the given image set whose proportion of the most frequent labels is equal to or greater than a predetermined accuracy as valid classes; and randomly extracting a predetermined number of images from each of the determined valid classes, wherein the pruned dataset is provided based on the predetermined number of images randomly extracted from the determined valid classes.

[0018] The steps to generate a pruned neural network can be performed without retraining the pruned neural network using the pruned dataset.

[0019] The method may further include: acquiring an image to be identified; and performing inference on the acquired image using a generated, pruned neural network.

[0020] In another general aspect, a neural network device includes: one or more processors configured to: acquire a first image set; pre-train a neural network based on the first image set; obtain weights for a plurality of layers of the pre-trained neural network; set a weight threshold for pruning the neural network to a target pruning rate based on a determined weight distribution; prune the plurality of layers of the neural network based on the weight threshold; predict changes in inference accuracy of the pruned plurality of layers of the neural network based on a pruned dataset input as a second image set; determine a current target layer among the plurality of layers of the neural network to be pruned using the weight threshold based on the predicted changes in inference accuracy; and generate a pruned neural network by pruning the determined current target layer.

[0021] The trimmed dataset can be a predetermined number of images randomly extracted from each class included in a given image set.

[0022] The one or more processors may also be configured to: determine the weight distribution of the plurality of layers of the neural network.

[0023] The device may further include: a memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform the acquisition of a first image set, pre-training, acquisition of weights, determination of the weight distribution of the multiple layers, setting of weight thresholds, prediction of changes in inference accuracy, determination of the current target layer, and pruning of the determined current target layer.

[0024] Regarding the determination of the current target layer, the one or more processors may be configured to: determine the layer, other than the previously pruned layers, that is predicted to have the lowest sensitivity to changes in the inference accuracy of the prediction as the current target layer.

[0025] For the prediction of changes in inference accuracy, the one or more processors may be configured to: calculate the sensitivity of each of the plurality of layers based on the difference between the inference accuracy before pruning each of the plurality of layers and the inference accuracy after pruning each of the plurality of layers.

[0026] The layer predicted to have the lowest sensitivity corresponds to the layer predicted to have the least impact on the reduction of inference accuracy of the neural network.

[0027] The one or more processors may be configured to set the weight value corresponding to the target pruning rate as a weight threshold when the determined weight distribution corresponds to a standard normal distribution.

[0028] The one or more processors may be configured to prune the current target layer by adjusting the pruning rate of the weights of the current target layer by updating a weight threshold until the inference accuracy of the neural network based on the pruned dataset decreases to the threshold accuracy.

[0029] The step of updating the weight threshold may include: increasing the current weight threshold when the inference accuracy of the neural network, which includes weights pruned to the current weight threshold, is not reduced to the threshold accuracy.

[0030] The one or more processors may be configured to: repeatedly perform the determination of the current target layer and the pruning of the determined current target layer until the target pruning rate is reached, or to complete the pruning of all layers of the neural network or a predetermined number of layers among all layers.

[0031] The trimmed dataset may include one of the following: an image set generated by randomly extracting a predetermined number of images for each class included in a given image set, and an image set generated by selecting valid classes from the trimmed dataset and randomly extracting a predetermined number of images for each selected valid class.

[0032] The one or more processors may be configured to provide a trimmed dataset by: randomly extracting a predetermined number of image samples from each class included in a given image set; performing inference on the randomly extracted samples using a neural network to determine a label corresponding to each sample in the randomly extracted samples; identifying the class whose proportion of the most frequent labels in the classes included in the given image set is equal to or greater than a predetermined accuracy as a valid class; and randomly extracting a predetermined number of images from each of the determined valid classes.

[0033] The one or more processors may also be configured to: acquire an image to be identified; and perform inference on the acquired image using a generated, pruned neural network.

[0034] The device can be any one of a personal computer, server device, mobile device, smartphone, autonomous vehicle, robot device, tablet device, augmented reality device, and Internet of Things device.

[0035] In another general aspect, a method for training a neural network for image recognition includes: acquiring a first image set; pre-training the neural network based on the first image set; obtaining weights for multiple layers of the pre-trained neural network; determining the weight distribution of the multiple layers of the neural network; determining an initial weight threshold based on the determined weight distribution; determining the sensitivity of each layer of the neural network based on a second image set, and selecting the layer of the neural network with the lowest sensitivity as the current target layer; and generating a pruned neural network by pruning weights in the current target layer that are less than or equal to the determined weight threshold.

[0036] The layer selected with the lowest sensitivity corresponds to the layer that is predicted to have the least impact on the reduction of inference accuracy of the neural network.

[0037] In another general aspect, a processor-implemented neural network method includes: setting a weight threshold for pruning a neural network to a target pruning rate based on a determined weight distribution; pruning multiple layers of the neural network based on the weight threshold; predicting changes in inference accuracy of the pruned multiple layers of the neural network based on an input pruned dataset; determining a current target layer among the multiple layers of the neural network to be pruned based on the predicted changes in inference accuracy; and generating a pruned neural network by pruning the determined current target layer.

[0038] A pruned dataset can be a predetermined number of data sources randomly extracted from each class included in a given dataset.

[0039] The method may further include: determining the weight distribution of the plurality of layers of the neural network.

[0040] The current target layer can be identified as the layer, excluding previously pruned layers, that is predicted to have the lowest sensitivity to changes in the inference accuracy of the prediction.

[0041] The step of predicting changes in inference accuracy may include: calculating the sensitivity of each of the plurality of layers based on the difference between the inference accuracy before pruning each layer and the inference accuracy after pruning each of the plurality of layers.

[0042] The layer predicted to have the lowest sensitivity corresponds to the layer predicted to have the least impact on the reduction of inference accuracy of the neural network.

[0043] The steps for setting the weight threshold may include: when the determined weight distribution corresponds to a standard normal distribution, setting the weight value corresponding to the target pruning rate as the weight threshold.

[0044] The steps for pruning the current target layer may include: adjusting the pruning rate of the weights of the current target layer by updating the weight threshold until the inference accuracy of the neural network based on the pruned dataset decreases to the threshold accuracy, thereby pruning the current target layer.

[0045] The step of updating the weight threshold may include: increasing the current weight threshold when the inference accuracy of the neural network, which includes weights pruned to the current weight threshold, is not reduced to the threshold accuracy.

[0046] The determination and pruning of the current target layer can be repeated until a certain number of layers or at least all of the multiple layers have been pruned.

[0047] A pruned dataset may include one of the following: a dataset generated by randomly extracting a predetermined number of data sources for each class included in a given dataset, or a dataset generated by selecting valid classes from the pruned dataset and randomly extracting a predetermined number of data sources for each selected valid class.

[0048] The steps of providing a pruned dataset may include: randomly extracting a predetermined number of samples from data sources from each class included in a given dataset; determining a label corresponding to each sample in the randomly extracted samples by performing inference on the randomly extracted samples using a neural network; identifying the classes in the given dataset that have a proportion of the most frequent labels equal to or greater than a predetermined accuracy as valid classes; and randomly extracting a predetermined number of data sources from each of the identified valid classes, wherein the pruned dataset is provided based on the predetermined number of data sources randomly extracted from the identified valid classes.

[0049] The steps to generate a pruned neural network can be performed without retraining the pruned neural network using the pruned dataset.

[0050] The method may further include: performing a generated, pruned neural network on the captured data.

[0051] In another general aspect, a neural network device includes: one or more processors configured to: set a weight threshold for pruning a neural network to a target pruning rate based on a determined weight distribution; prune multiple layers of the neural network based on the weight threshold; predict changes in inference accuracy of the pruned multiple layers of the neural network based on an input pruned dataset; determine a current target layer among the multiple layers of the neural network to be pruned using the weight threshold based on the predicted changes in inference accuracy; and generate a pruned neural network by pruning the determined current target layer.

[0052] A pruned dataset can be a predetermined number of data sources randomly extracted from each class included in a given dataset.

[0053] The one or more processors may also be configured to: determine the weight distribution of the plurality of layers of the neural network.

[0054] The device may further include: a memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform the determination of the weight distribution of the plurality of layers, the setting of weight thresholds, the prediction of changes in inference accuracy, the determination of the current target layer, and the pruning of the determined current target layer.

[0055] Regarding the determination of the current target layer, the one or more processors may be configured to: determine the layer, other than the previously pruned layers, that is predicted to have the lowest sensitivity to changes in the inference accuracy of the prediction as the current target layer.

[0056] For the prediction of changes in inference accuracy, the one or more processors may be configured to: calculate the sensitivity of each of the plurality of layers based on the difference between the inference accuracy before pruning each of the plurality of layers and the inference accuracy after pruning each of the plurality of layers.

[0057] The layer predicted to have the lowest sensitivity corresponds to the layer predicted to have the least impact on the reduction of inference accuracy of the neural network.

[0058] The one or more processors may be configured to set the weight value corresponding to the target pruning rate as a weight threshold when the determined weight distribution corresponds to a standard normal distribution.

[0059] The one or more processors may be configured to prune the current target layer by adjusting the pruning rate of the weights of the current target layer by updating a weight threshold until the inference accuracy of the neural network based on the pruned dataset decreases to the threshold accuracy.

[0060] The weight threshold can be updated by increasing the current weight threshold when the inference accuracy of the neural network, which includes weights pruned to the current weight threshold, has not been reduced to the threshold accuracy.

[0061] The determination of the current target layer and the pruning of the determined current target layer can be repeatedly performed by one or more processors until a certain number of layers or at least all of the multiple layers have been pruned.

[0062] A pruned dataset may include one of the following: a dataset generated by randomly extracting a predetermined number of data sources for each class included in a given dataset, or a dataset generated by selecting valid classes from the pruned dataset and randomly extracting a predetermined number of data sources for each selected valid class.

[0063] The one or more processors may be configured to provide a pruned dataset by: randomly extracting a predetermined number of samples from data sources from each class included in a given dataset; performing inference on the randomly extracted samples using a neural network to determine a label corresponding to each sample in the randomly extracted samples; identifying the classes in the given dataset that have a proportion of the most frequent labels equal to or greater than a predetermined accuracy as valid classes; and randomly extracting a predetermined number of data sources from each of the identified valid classes.

[0064] The device can generate a pruned neural network based on the captured data.

[0065] The device may be any one of a personal computer (PC), server device, mobile device, smartphone, autonomous vehicle, robotic device, tablet device, augmented reality (AR) device, and Internet of Things (IoT) device, and the one or more processors are configured to perform either speech recognition or image recognition using the generated pruned neural network.

[0066] In one general aspect, a processor-implemented method includes: determining the weight distribution of multiple layers of a neural network; determining an initial weight threshold based on the determined weight distribution; determining the sensitivity of each layer of the neural network and selecting the layer of the neural network with the lowest sensitivity as the current target layer; and generating a pruned neural network by pruning the weights in the current target layer that are less than or equal to the determined weight threshold.

[0067] The layer selected with the lowest sensitivity corresponds to the layer that can be predicted to have the least impact on the reduction of inference accuracy of the neural network.

[0068] Other features and aspects will become clear from the following detailed description, the accompanying drawings, and the claims. Attached Figure Description

[0069] Figure 1 Examples of neural networks according to one or more embodiments are shown.

[0070] Figure 2A and Figure 2B Examples of neural network pruning according to one or more embodiments are shown.

[0071] Figure 3 A block diagram illustrating an example hardware configuration of a neural network device according to one or more embodiments.

[0072] Figure 4 An example of a trimmed dataset according to one or more embodiments is shown.

[0073] Figures 5A to 5CThe types of trimmed datasets according to one or more embodiments are shown.

[0074] Figure 5D Show preparation Figure 5C An example of processing a trimmed dataset.

[0075] Figure 6 This is an example of a graph illustrating the setting of weight thresholds for neural network pruning, according to one or more embodiments.

[0076] Figure 7 An example flowchart illustrating the process of performing neural network pruning according to one or more embodiments is shown.

[0077] Figure 8 An example flowchart illustrating a method for performing neural network pruning according to one or more embodiments is shown.

[0078] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0079] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0080] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the stated features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0081] Throughout the specification, when an element such as a layer, region, or substrate is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0082] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0083] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0084] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, upon understanding the disclosure of this application. Unless expressly defined herein, terms (such as those defined in general dictionaries) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in the disclosure of this application, and shall not be interpreted in an idealized or overly formalistic manner.

[0085] Figure 1 An example of a neural network 100 according to one or more embodiments is shown.

[0086] For example, the automation of pattern recognition or analysis techniques has been achieved through neural network models implemented as processors in dedicated computing architectures. After extensive training, these models can provide a computationally intuitive mapping between input and output patterns, or pattern recognition of the input pattern. The ability to generate such a mapping or perform such pattern recognition during training is referred to as the learning capability of the neural network. This training capability also enables dedicated computing architectures to classify such input patterns or portions of input patterns as belonging to one or more predetermined groups. Furthermore, due to specialized training, such specially trained neural networks can thus possess, for example, the generalization ability to generate relatively accurate or reliable outputs for input patterns that the neural network might not otherwise have been trained on.

[0087] Reference Figure 1 As a non-limiting example, neural network 100 may be a deep neural network (DNN). A DNN may include multiple layers. For example, a deep neural network may include an input layer, an output layer, and multiple hidden layers, with input data applied to the input layer, an output layer for outputting a result obtained through predictions based on training and input data, and multiple hidden layers for performing neural network operations between the input and output layers.

[0088] In such an example, depending on the algorithm used to process the information, the DNN can be one or more of fully connected networks, convolutional neural networks, recurrent neural networks, etc., or may include different or overlapping neural network parts having such fully connected, convolutional, or recurrent connections respectively. The neural network 100 can be configured to perform object classification, object recognition, speech recognition, and image recognition, as non-limiting examples, by mapping input data and output data to each other in a non-linear relationship based on deep learning. As a non-limiting example, such deep learning refers to a machine learning scheme implemented by a processor for solving a problem (such as a problem related to automatic image or speech recognition) based on a dataset. It should be noted here that the term "may" (e.g., what an example or embodiment may include or implement) is used with respect to examples or embodiments to indicate that there exists at least one example or embodiment that includes or implements such a feature, while all examples and embodiments are not limited thereto.

[0089] In one example embodiment, neural network 100 may be a neural network for image recognition. Neural network 100 may receive a first image set (e.g., training images), perform pre-training using the first image set, and obtain the weights of multiple layers of the neural network. To reduce the complexity of neural network 100 and minimize excessive memory allocation, pruning may be performed on neural network 100 after pre-training. Various training methods can be used to implement the pre-training of the neural network for image recognition according to embodiments of the present invention, and the present invention is not limited thereto.

[0090] In the following example, the input layer may correspond to the lowest layer of the neural network, and the output layer may correspond to the highest layer. The layer order can be assigned and named sequentially from the output layer, which is the highest layer, to the input layer, which is the lowest layer. For example, hidden layer 2 may correspond to the layer above hidden layer 1 and the input layer, but below the output layer.

[0091] In a DNN that includes multiple layers, a relatively higher layer adjacent to a lower layer can receive a value obtained by multiplying the output value of the relatively lower layer by a weight and applying a bias to it, and output a specific operational value that can be applied to another higher layer adjacent to the higher layer in a similar manner (e.g., as input).

[0092] The method of training neural networks is called deep learning, and as mentioned above, deep learning can use various algorithmic methods (such as in convolutional neural networks or recurrent neural networks).

[0093] Training a neural network can be represented by determining and updating weights and biases between multiple layers or between multiple nodes (or neurons) belonging to different layers that are adjacent to each other. However, this reference to "neuron" is not intended to reveal any relevance to how neural network architecture computationally maps or thereby intuitively identifies information that is similar to how human neurons operate. In other words, the term "neuron" is simply a specialized term to refer to a node in a hardware-implemented neural network and will have the same meaning as a node in a neural network.

[0094] For example, the weights and biases between layers or multiple layers or multiple neurons can be collectively referred to as the connectivity of a neural network. Therefore, training a neural network can be represented as establishing and training connectivity.

[0095] Reference Figure 1 Each of the multiple layers 101 to 107 may include multiple nodes.

[0096] Reference Figure 1 In this context, multiple nodes in one layer and multiple nodes in adjacent layers are fully connected. For example, node 3-1 of hidden layer 2 105 is connected to all nodes in hidden layer 1 103 (i.e., nodes 2-1 to 2-4) and receives inputs that are obtained by multiplying the output values ​​of each node by specific weights.

[0097] The data input to input layer 101 (e.g., image data) is processed through multiple hidden layers, so that the final output value 109 (e.g., label) is output from output layer 107. In this state, as the weights increase, the connectivity between the corresponding two nodes is strengthened, and as the weights decrease, the connectivity between the two nodes is weakened. For example, the weights can have values ​​between 0 and 1, and when the weights are 0, there is no connectivity between the two nodes. However, the numerical range of the weights is not limited to this and can vary depending on the implementation of the neural network (e.g., based on various ranges and activation functions).

[0098] As connectivity is increased through weights, the connectivity of a neural network can be strengthened, but its complexity can also increase. Consequently, the amount of memory allocated to storing the weights can increase, thus reducing the overall performance and efficiency of the neural network. Therefore, compression methods are beneficial for reducing the size of neural networks, decreasing system cost, and reducing the amount of computation performed in the implementation of neural networks while maintaining their performance.

[0099] Figure 2A and Figure 2B An example of pruning a neural network 210 according to one or more embodiments is shown.

[0100] Reference Figure 2A Neural network 210 corresponds to the neural network before pruning, and neural network 220 corresponds to the neural network after pruning. Pruning can support and provide functionality for performing both coarse-grained neural network pruning (e.g., pruning channels, kernels, or nodes from a neural network model) and more targeted fine-grained neural network pruning (e.g., pruning individual weights from a neural network model).

[0101] In one example, connections or relationships are formed between all pairs of neurons or node combinations included in two different adjacent layers of neural network 210. Specifically, since neural network 210 is a fully connected neural network, the weights representing the connection strength between two neurons belonging to different adjacent layers included in neural network 210 can be greater than 0. Therefore, when connectivity exists between neurons in all adjacent layers, the complexity of the entire neural network increases, and thus the accuracy and reliability of the neural network's predictions can decrease due to overfitting.

[0102] To reduce the complexity of a neural network, pruning can be performed on it.

[0103] For example, such as Figure 2A As shown, when the weights between connected nodes in neural network 210 are less than or equal to a specific threshold, neural network pruning can represent compressing or removing the connectivity between nodes. For example, when the weights between node 1 and nodes 2-3 in neural network 210 are less than or equal to the specific threshold, the pruning process can set the weights between node 1 and nodes 2-3 in neural network 210 to 0, thereby removing the connectivity between node 1 and nodes 2-3, as shown in the pruned neural network 220. Similarly, in the pruned neural network 220, the connectivity between some nodes in neural network 210 can be weakened or removed.

[0104] Each layer of a neural network can be analyzed to determine the nodes and node connections suitable for pruning. In this state, unless the inference accuracy of the neural network (i.e., the output of the neural network) is excessively reduced, the layers to be pruned and the weights associated with those nodes can be determined.

[0105] When pruning is performed between the weights of layers included in a neural network, connections between neurons with weights less than a predefined weight threshold can be identified, and connections between neurons with such weights can be removed or weakened.

[0106] Reference Figure 2BThis shows the kernel for the k-th kernel, which is the n-th layer (convolutional layer) in a neural network. n_k 230 (where n and k are natural numbers greater than 0) is the result of the pruning process. For example, as a setting for neural network pruning, the weight threshold τ is assumed to be 0.5.

[0107] In one example, the weight threshold can be a layer-wise weight threshold, calculated based on the statistical distribution of the fully dense weights in each layer, and weight pruning can be performed to mask those weights that are less than the corresponding layer-specific threshold. In other examples, a single weight threshold can be defined for the entire network. In some examples, layer-specific weight thresholds can improve the speed of pruning and the accuracy of the resulting pruned network.

[0108] Before trimming, KERNEL n_k 230 can have various weight values. During the pruning process, in KERNEL n_k Of the 230 weights, all weights less than or equal to the weight threshold (τ = 0.5) can be pruned to 0, and the pruned KERNEL n_k 240 can be generated. Thus, with KERNEL n_k Compared to 230, the pruned KERNEL n_k 240 can weaken the connectivity between adjacent nodes. Therefore, due to the weight of 0, the amount or number of computational operations between adjacent nodes can be reduced.

[0109] Figure 3 A block diagram illustrating the hardware configuration of a neural network device 100 according to one or more embodiments is shown.

[0110] Reference Figure 3 The neural network device 100 may include a processor 110 and a memory 120. In one example, the neural network device 100 may also (e.g., in memory 120) store instructions that, when executed by the processor 110, configure one or more operations implemented therein by the processor 110. The processor 110 and memory 120 may respectively represent one or more processors 110 and one or more memories 120. Figure 3 The neural network device 100 shown includes example elements relevant to this example. Therefore, the neural network device 100 may also include, in addition to... Figure 3 Other elements or components besides the ones shown are illustrated.

[0111] The neural network device 100 corresponds to a computing device that includes the pruning of a neural network. For example, as a non-limiting example, the neural network device 100 may be a personal computer (PC), a server, a mobile device, etc., and may also correspond to or be a device installed in an autonomous vehicle, a robot, a smartphone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, and similar devices that can perform speech recognition and image recognition by implementing a neural network. However, this disclosure is not limited to this and may correspond to various types of devices.

[0112] Processor 110 is a hardware configuration for performing general control functions to control the operation of neural network device 100. For example, processor 110 may typically control neural network device 100 by executing instructions stored in memory 120 of neural network device 100. As a non-limiting example, processor 110 may be implemented as a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), neural processor (NPU), etc., disposed in neural network device 100.

[0113] The memory 120, which serves as hardware for storing data from various neural networks processed by the processor 110, can store parameters of the neural network, such as multiple pruning data points about the neural network and datasets to be input into the neural network. Furthermore, the memory 120 can store various applications that will be driven by the processor 110 (e.g., applications for neural network pruning, neural network driving applications, drivers, etc.).

[0114] The memory 120 may include at least one of volatile memory and non-volatile memory. Non-volatile memory may include: read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memory may include: dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), etc. Furthermore, the memory 120 may include at least one of hard disk drive (HDD), solid-state drive (SSD), compact flash memory (CF) card, secure digital card (SD) card, micro-secure digital card (Micro-SD) card, mini-secure digital card (Mini-SD) card, extreme digital card (xD) card, and Memory Stick.

[0115] Refer again Figure 3The processor 110 sets weight thresholds for pruning the neural network to a target pruning rate based on the weight distribution of the layers included in the neural network. In one example, assuming the weight distribution corresponds to a Gaussian distribution like a standard normal distribution, the processor 110 can set the weight values ​​corresponding to the target pruning rate as the weight thresholds. However, the processor 110 can set weight thresholds suitable for neural network pruning by using various statistical distributions other than the standard normal distribution. For example, the weight thresholds can be determined based on statistics describing the weight values ​​within a layer (e.g., the mean, median, mode, standard deviation, variance, etc. of the layer's weight values).

[0116] Processor 110 performs neural network pruning by repeatedly measuring and verifying the accuracy of the neural network's inferences on the pruned dataset by inputting the pruned dataset into the neural network. The preparation of the pruned dataset is described in detail with reference to the accompanying figures below.

[0117] Processor 110 can predict or determine the impact of pruning each layer of a neural network based on weight thresholds on changes in the inference accuracy (e.g., image recognition accuracy) of the neural network using a prepared pruned dataset (also referred to as a second image set). The prediction of the impact can be performed by calculating the sensitivity for each layer based on the difference between the inference accuracy before and after pruning. Sensitivity indicates the degree to which the output of the neural network is affected by the inputs and / or weight perturbations of the neural network. A sensitivity measurement can be defined as the mathematical expectation of the output deviation due to a desired input bias over a continuous interval for the entire input pattern, and can be used to determine the output's response to the model when various input parameters are introduced into the model.

[0118] Processor 110 can determine, based on the predicted impact, the current target layer among multiple layers in the neural network that will be pruned using a weight threshold. Processor 110 can identify the layer predicted to have the lowest sensitivity among the layers other than those previously pruned as the current target layer. In other words, the layer predicted to have the lowest sensitivity can represent the layer predicted to have the least impact on reducing the inference accuracy of the neural network.

[0119] When the current target layer is determined, the processor 110 prunes the determined current target layer. Specifically, the processor 110 can adjust the pruning rate of the weights of the current target layer by updating the weight threshold until the inference accuracy of the neural network using the pruned dataset is determined to have decreased to the threshold accuracy, in order to prune the current target layer.

[0120] After the current target layer is pruned, the processor 110 can then search for target layers to be pruned in other layers of the neural network, and can repeatedly perform pruning on newly found target layers, thereby pruning the entire neural network. The pruning of the entire neural network is determined to be complete when the neural network pruning has reached the initially set target pruning rate, or when pruning has been completed on all layers or a predetermined number of layers included in the neural network. The processor 110 can then use the pruned neural network to perform image recognition.

[0121] Figure 4 An example of a trimmed dataset 420 according to one or more embodiments is shown.

[0122] Reference Figure 4 The pruning dataset 420 can be a dataset (e.g., an image set) used to measure and verify changes in the inference accuracy of the neural network (e.g., sensitivity of each layer) when pruning is performed on the neural network.

[0123] The trimmed dataset 420 can be prepared from the initial given dataset (e.g., a given set of images) 410. In this example, as a non-limiting example, the given dataset 410 can be a public dataset (such as the ImageNet database, a modified National Institute of Standards and Technology (MNIST) database, etc.). However, various other types of databases can be implemented.

[0124] The dataset 420 can be pruned to reduce or avoid overfitting to the given dataset 410 when it is used as is, and to shorten the pruning time of the neural network. However, the preparation process for pruning the dataset 420 can vary depending on the type of the given dataset 410. Traditionally, overfitting may be due to bias or overtraining, which can lead to a decrease in the performance of the neural network. By pruning the neural network, unnecessary parameters that can cause such overfitting can be removed or mitigated. Therefore, the final training and the performance of the neural network can be enhanced through neural network pruning.

[0125] Figures 5A to 5C The types of trimmed datasets according to one or more embodiments are shown.

[0126] Reference Figure 5AThe pruned dataset 512 (the first type of pruned dataset) may correspond to the same dataset as the given dataset (e.g., a given set of images) 511. As described above, in neural network pruning, when it is desired that no overfitting problem occurs when using the given dataset 511, the same dataset as the given dataset 511 may be prepared as the pruned dataset 512. For example, when the given dataset 511 corresponds to the ImageNet database, the first type of pruned dataset may be a data source (e.g., images) of millions to tens of millions of images in the ImageNet database.

[0127] Reference Figure 5B A trimmed dataset 522 (a second type of trimmed dataset) can be prepared by randomly extracting a specific number of data sources (e.g., images) from each of the various classes included in a given dataset (e.g., a given image set) 521. For example, the given dataset 521 can be assumed (i.e., as a non-limiting example) to be a dataset (e.g., an image set) obtained from the ImageNet database, which, as an example, includes over one million data sources with approximately 1000 classes. In this example, the trimmed dataset 522 could be a new dataset (e.g., an image set) that only includes data sources obtained by randomly extracting approximately 1 to 10 data sources (e.g., images) from each of the approximately 1000 classes in the given dataset 521. The number of data sources to be randomly extracted for each class can be arbitrarily set and can vary for each class. Figure 5B The preparation method described herein can reduce the size of the dataset, thus shortening the neural network pruning processing time and preventing the phenomenon of the pruned neural network overfitting only to a specific database.

[0128] This can be achieved by using each class from the known classes in the ImageNet database. Figure 5B The second type of pruning dataset provided in the method described is used to perform pruning on typical neural networks (such as "Inception-v3").

[0129] However, since the inference algorithms of neural networks (such as the “Aic classifier”) trained with their own unknown datasets are not recognized, it is necessary to select classes that can be effectively inferred by neural networks (such as the “Aic classifier”) from the classes already defined in the ImageNet database.

[0130] Reference Figure 5C Given a dataset 531, a neural network trained with its own unknown dataset (such as an "Aic classifier") can be provided. Figure 3 The processor 110 can generate a trimmed dataset 533 (a third type of trimmed dataset) according to different preparation methods.

[0131] Specifically, given a dataset 531, it is input into a neural network, and processor 110 can perform inference (e.g., image recognition) on data sources (e.g., images) of the given dataset 531 by implementing the neural network. As a result of the neural network's inference, processor 110 can determine the label of each data source in each class of the given dataset 531. The label can be an identifier indicating the class to which the data source belongs. Processor 110 selects classes with accuracy exceeding a certain level and determines the selected classes as valid classes 532 (e.g., class A1, class B2, class D1, class D3, ..., class Z51). Finally, processor 110 can randomly extract a certain number of data sources (e.g., 1 to 10 data sources) from each valid class 532 and prepare a new dataset that includes only the extracted data sources as a pruned dataset 533. However, as Figure 5B As described above, the number of data sources randomly extracted for each valid class 532 can be set arbitrarily and can be changed for each class.

[0132] Figure 5D The preparation is shown in detail. Figure 5C Processing of the pruned dataset (the third type of pruned dataset).

[0133] Reference Figure 5D Given a dataset, it is assumed to include a total of 1000 classes, from class 1 to class 1000. Processor 110 may randomly extract 100 samples from data sources for each class. Processor 110 may then use a neural network to perform inference (e.g., image recognition) using each data source (e.g., an image) as input and determine a label associated with each data source. The label may be an identifier indicating the class to which the data source belongs.

[0134] Processor 110 can identify valid classes as those whose proportion of the most frequent labels (i.e., the labels that appear most often in the results) exceeds a certain accuracy (or proportion) (e.g., 60%). Therefore, processor 110 can select multiple classes (class 1, class 3, ..., class 999) that meet the above conditions as valid classes.

[0135] Processor 110 can randomly extract a specific number (e.g., 1 to 50) of data sources from each valid class and prepare a new dataset containing only the extracted data sources as a third-type pruned dataset. The number of samples to be randomly extracted for each class, the value of the specific accuracy used to select valid classes, and the number of data sources to be randomly extracted from valid classes can be arbitrarily set and can vary for each class.

[0136] when Figure 3 The processor 110 of the neural network device 100 uses... Figures 5A to 5D When preparing a pruned dataset using the method described in the document, the measurement and verification of the inference accuracy in the pruning process of the neural network can be performed.

[0137] Figure 6 This is a graph illustrating the settings of weight thresholds for neural network pruning, according to one or more embodiments.

[0138] Reference Figure 6 The processor 110 sets a weight threshold for pruning the neural network to a target pruning rate based on the weight distribution of the layers included in the neural network. In this state, assuming the weight distribution corresponds to a Gaussian distribution (such as a standard normal distribution), the processor 110 can set the weight values ​​corresponding to the target pruning rate as the weight threshold.

[0139] Equation 1 below calculates the weighted threshold τ corresponding to the target pruning rate (α%).

[0140] Equation 1:

[0141] τ=(Z·σ)+μ

[0142] Specifically, first, processor 110 obtains the Z-value corresponding to the target pruning rate (α%) based on a standard normal distribution with respect to the weight distribution assumptions included in the layers of the neural network. The correspondence between the target pruning rate (α%) and the Z-value can be predetermined. Then, processor 110 calculates the weight threshold τ based on the mean μ and standard deviation σ of the weights.

[0143] For example, when the target pruning rate (α%) is 30%, Z = 0.39 can be calculated. The mean μ and standard deviation σ of the weights can be calculated separately, so the weight threshold τ used for neural network pruning can be set through it.

[0144] However, the processor 110 can set weight thresholds suitable for neural network pruning based on various other statistical distributions besides the standard normal distribution.

[0145] Figure 7 A flowchart illustrating the process of performing neural network pruning according to one or more embodiments is shown. Although the order of some operations or the omission of some operations may be changed or omitted without departing from the spirit and scope of the illustrative examples described, Figure 7 The operations can be performed in the order and manner shown. Figure 7 Many of the operations shown can be performed in parallel or simultaneously. Figure 7 One or more boxes, and combinations thereof, may be implemented by a computer based on dedicated hardware, or a combination of dedicated hardware and computer instructions, that performs the specified function. In addition to the following... Figure 7 In addition to the description, Figures 1 to 6 The description also applies to Figure 7 And it is incorporated here by reference. Therefore, the above description need not be repeated here.

[0146] Reference Figure 7 In operation 701, processor 110, based on the weight distribution of layers included in the neural network, uses... Figure 6 The method described herein is used to set weight thresholds for pruning the neural network to a target pruning rate.

[0147] In operation 702, processor 110 calculates the sensitivity of each layer of the neural network and determines the layer with the lowest sensitivity as the current target layer. In one example, the layer determined to have the lowest sensitivity may correspond to the layer predicted to have the least impact on the reduction in inference accuracy of the neural network pruned by the neural network. Processor 110 can use the above... Figures 5A to 5D Sensitivity is calculated using at least one of the pruning datasets described in the document, and the impact of pruning each layer based on a weight threshold on the inference accuracy of the neural network can be predicted by using the pruning dataset.

[0148] The sensitivity can be calculated using Equation 2 below.

[0149] Equation 2:

[0150] Make

[0151]

[0152] Referring to Equation 2, τ is the weight threshold, and x is the weight. t represents the order in which layers of the neural network are pruned, and k represents the index of the current target layer to be pruned. For example, t = 3 when the current target layer to be pruned corresponds to the third layer to be pruned. W t This represents the weight of a position in sequence t. Let P(W) represent the weights of the k-th layer pruned sequentially by t with respect to τ. t )and They respectively represent the use of W t The inference accuracy and usage of neural networks The inference accuracy of the neural network.

[0153] In other words, according to Equation 2, sensitivity for each layer can be calculated based on the difference between the inference accuracy before pruning each layer and the inference accuracy after pruning each layer.

[0154] In operation 703, processor 110 prunes weights in the current target layer that are less than or equal to the weight threshold τ.

[0155] In operation 704, processor 110 measures the inference accuracy of the neural network for the pruned dataset based on the pruning results of the current target layer.

[0156] In operation 705, processor 110 compares the measured inference accuracy with a threshold accuracy and determines whether the measured inference accuracy is lower than the threshold accuracy. If the measured inference accuracy is lower than the threshold accuracy, operation 707 is executed. However, if the measured inference accuracy is not lower than the threshold accuracy, operation 706 is executed.

[0157] In operation 706, when the inference accuracy based on pruning the current target layer using the weight threshold τ does not decrease to the threshold accuracy, processor 110 updates the weight threshold τ by increasing the weight threshold τ. For example, processor 110 updates the weight threshold τ by increasing the weight threshold τ by δ, such as τ = τ + δ. In one example, δ can be an arbitrarily set value based on various factors (such as the weight distribution of the neural network, the pruning rate of the current target layer, etc.).

[0158] Processor 110 performs operations 703 to 705 again based on the updated weight threshold. In other words, processor 110 updates the weight threshold by adjusting the pruning rate of the weights of the current target layer until the inference accuracy of the neural network for the pruned dataset decreases to the threshold accuracy, thereby pruning the current target layer.

[0159] Operations 703 through 706 can be performed using the following equation 3.

[0160] Equation 3:

[0161]

[0162]

[0163] Referring to Equation 3, β represents the threshold accuracy, and as mentioned above, when the decrease in inference accuracy is less than or equal to a certain level β, pruning can be performed on the current target layer, while increasing the pruning rate of the current target layer by increasing the weight threshold τ.

[0164] As a result of operation 705, when the inference accuracy of the measurement is less than the threshold accuracy, processor 110 terminates the pruning of the current target layer and executes operation 707.

[0165] In operation 707, processor 110 determines whether pruning of all layers of the neural network has been completed. If it is determined that pruning of all layers has not been completed, operation 708 is executed.

[0166] In operation 708, processor 110 stores in memory 120 the result of the pruning performed on the current target layer just before, and repeats the operation from operation 702 to perform pruning on the next sequential (t+1) layer.

[0167] As a result of operation 707, processor 110 terminates neural network pruning when pruning of all layers is complete.

[0168] Referring again to operation 707, the neural network pruning process terminates when pruning of all layers has been completed. However, the example is not limited to this, and according to another example, processor 110 may determine whether the initially set target pruning rate has been reached as a condition for terminating the neural network pruning process. In other words, in operation 707, as a result of repeated pruning of the neural network layers, processor 110 may control the termination of the neural network pruning process when the initially set target pruning rate is reached.

[0169] In another example, processor 110 may determine that the conditions for terminating neural network pruning are met when the target pruning rate is reached or pruning of all layers is completed.

[0170] Typically, retraining of neural networks is repeatedly performed to mitigate the decrease in accuracy caused by pruning. However, when retraining a neural network is performed, various resources (e.g., time, processing resources, etc.) may be required. Therefore, as a simulation for pruning the Inception-v3 model by 57%, a timeframe of over 7 days can be utilized when using 6 GPUs, and 2 or 3 days can be utilized for ResNet-101, which is the fastest model.

[0171] According to the neural network pruning method performed by the neural network device 100 according to the disclosed example, efficient pruning of the neural network can be achieved because the pruning of the neural network is performed without retraining the neural network, for example, using the pruned dataset. The neural network pruning method can be performed as an example final stage of training (e.g., after training and without retraining), and / or performed by a neural network device (such as a smartphone) for inference. One or more embodiments of the neural network device can be configured to reduce the computational cost of processing the neural network, thereby advantageously improving the computational speed of one or more embodiments of the neural network device compared to typical neural network devices, thus solving such technical problems and providing technical improvements.

[0172] Figure 8 A flowchart illustrating a method for pruning a neural network according to one or more embodiments is provided. Although the order of some operations or the omission of some operations may be changed or omitted without departing from the spirit and scope of the illustrative examples described, Figure 8The operations can be performed in the order and manner shown. Figure 8 Many of the operations shown can be performed in parallel or simultaneously. Figure 8 One or more boxes, and combinations thereof, may be implemented by a computer based on dedicated hardware, or a combination of dedicated hardware and computer instructions, that performs the specified function. In addition to the following... Figure 8 In addition to the description, Figures 1 to 7 The description also applies to Figure 8 And it is incorporated here by reference. Therefore, the above description need not be repeated here. Because Figure 8 The method is related to the example described in the above figures, therefore the description given above but omitted here can be applied to... Figure 8 The method.

[0173] In operation 801, processor 110 sets a weight threshold for pruning the neural network to a target pruning rate based on the weight distribution of the layers included in the neural network.

[0174] In operation 802, processor 110 predicts the impact of pruning each layer based on a weight threshold on the inference accuracy of the neural network by using a pruned dataset.

[0175] In operation 803, processor 110 determines the current target layer in the layers included in the neural network that will be pruned using weight thresholds based on the predicted impact.

[0176] In operation 804, processor 110 prunes the determined current target layer.

[0177] Once the pruning of the current target layer is complete, the processor 110 repeatedly performs pruning on another layer of the neural network.

[0178] Table 1 below shows simulation results comparing the accuracy of the typical pruning method with that of the example embodiment pruning method, in which pruning is performed without, for example, retraining. The simulation results of the example embodiment are obtained by pruning the Inception-v3 neural network using a total of 1000 data sources prepared by randomly extracting one data source from each of the 1000 classes in the ImageNet database.

[0179] Table 1:

[0180]

[0181] The pre-pruned accuracy of the original Inception-v3 neural network in one or more embodiments herein is 77.97%. As described above, according to simulations, the accuracy is significantly reduced in a typical method that performs pruning without retraining, compared to the method of the example embodiment which only has an accuracy reduction of about 1.78%. Therefore, in the pruning method according to one or more embodiments herein, a high rate of pruning can be performed while reducing accuracy loss, even when the neural network is pruned without retraining.

[0182] Regarding Figures 1 to 8The described neural network device, neural network pruning device 100, processor 110, memory 120, and other devices, units, modules, apparatuses, and other components are implemented by hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component or two or more hardware components. The hardware components can have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0183] Perform as described in this application and Figures 1 to 8The methods of operation illustrated herein are executed by computing hardware (e.g., by one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation or two or more operations may be executed by a single processor or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations (e.g., respective operations of the methods implemented as processors) may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation or two or more operations.

[0184] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above are written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers, such as machines or special-purpose computers, to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing operations performed by the hardware components and methods described above.

[0185] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R, BD-RTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as, multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD) storage devices, magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other devices configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and to provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers, enabling said one or more processors or computers to execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0186] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents. Therefore, the scope of the disclosure is not limited by the specific embodiments but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents should be construed as included in the disclosure.

Claims

1. A method for training a neural network for image recognition, comprising: Obtain the first image set; The neural network is pre-trained based on the first image set; Obtain the weights of multiple layers in the pre-trained neural network; Based on a defined weight distribution, a weight threshold is set to prune the neural network to the target pruning rate. The multiple layers of the neural network are pruned based on weight thresholds; The variation in inference accuracy of multiple layers of a neural network for predicting the trimming of images is based on the trimming dataset as a second image set as input. Based on the change in the inference accuracy of the prediction, determine the current target layer to be pruned among the plurality of layers of the neural network; and A pruned neural network is generated by pruning the current target layer. The step of pruning the current target layer includes: adjusting the pruning rate of the weights of the current target layer by updating the weight threshold until the inference accuracy of the neural network based on the pruned dataset decreases to the threshold accuracy, thereby pruning the current target layer.

2. The method according to claim 1, wherein, The trimmed dataset is a predetermined number of images randomly extracted from each class included in a given image set.

3. The method according to claim 1, further comprising: Determine the weight distribution of the multiple layers of the neural network.

4. The method according to claim 1, wherein, The current target layer is identified as the layer, excluding previously pruned layers, that is predicted to have the lowest sensitivity to changes in the inference accuracy of the prediction.

5. The method according to claim 4, wherein, The step of predicting changes in inference accuracy includes: calculating the sensitivity of each of the plurality of layers based on the difference between the inference accuracy before pruning each of the plurality of layers and the inference accuracy after pruning each of the plurality of layers.

6. The method according to claim 4, wherein, The layer predicted to have the lowest sensitivity corresponds to the layer predicted to have the least impact on the reduction of inference accuracy of the neural network.

7. The method according to claim 1, wherein, The steps for setting the weight threshold include: when the determined weight distribution corresponds to a standard normal distribution, setting the weight value corresponding to the target pruning rate as the weight threshold.

8. The method according to claim 1, wherein, The steps for updating the weight threshold include: increasing the current weight threshold when the inference accuracy of the neural network, which includes weights pruned to the current weight threshold, is not reduced to the threshold accuracy.

9. The method according to claim 1, wherein, Repeat the process of determining the current target layer and pruning the determined current target layer until the target pruning rate is reached, or the pruning of all layers of the neural network or a predetermined number of layers is completed.

10. The method according to claim 1, further comprising: Provide a trimmed dataset. The trimmed dataset includes one of the following: an image set generated by randomly extracting a predetermined number of images for each class included in a given image set, and an image set generated by selecting valid classes from the trimmed dataset and randomly extracting a predetermined number of images for each selected valid class.

11. The method according to claim 10, wherein, The steps to provide a pruned dataset include: A predetermined number of images are randomly extracted from each class included in a given image set; By using a neural network to perform inference on randomly selected samples, the label corresponding to each sample in the randomly selected samples is determined; The classes included in a given image set with the proportion of their most frequent labels equal to or greater than a predetermined accuracy are identified as valid classes; and Randomly extract a predetermined number of images from each of the determined valid classes. The pruned dataset is provided based on a predetermined number of images randomly extracted from a defined valid class.

12. The method according to claim 1, wherein, Perform the steps to generate a pruned neural network without retraining the pruned neural network using the pruned dataset.

13. The method according to claim 1, further comprising: Acquire the image to be identified; and The generated, pruned neural network is used to perform inference on the acquired image.

14. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 13.

15. A neural network device for image recognition, comprising: One or more processors are configured as follows: Obtain the first image set; The neural network is pre-trained based on the first image set; Obtain the weights of multiple layers in the pre-trained neural network; Based on a defined weight distribution, a weight threshold is set to prune the neural network to the target pruning rate. The multiple layers of the neural network are pruned based on weight thresholds; The variation in inference accuracy of multiple layers of a neural network for predicting the trimming of images is based on the trimming dataset as a second image set as input. Based on the change in the inference accuracy of the prediction, determine the current target layer among the plurality of layers of the neural network that will be pruned with a weight threshold; and A pruned neural network is generated by pruning the current target layer. In order to prune the current target layer, the one or more processors are configured to: adjust the pruning rate of the weights of the current target layer by updating a weight threshold until the inference accuracy of the neural network based on the pruned dataset decreases to the threshold accuracy, thereby pruning the current target layer.

16. The neural network device according to claim 15, wherein, The trimmed dataset is a predetermined number of images randomly extracted from each class included in a given image set.

17. The neural network device according to claim 15, wherein, The one or more processors are further configured to: determine the weight distribution of the plurality of layers of the neural network.

18. The neural network device of claim 15, further comprising: The memory stores instructions that, when executed by the one or more processors, configure the one or more processors to perform the acquisition of a first image set, pre-training, acquisition of weights, determination of the weight distribution of the multiple layers, setting of weight thresholds, prediction of changes in inference accuracy, determination of the current target layer, and pruning of the determined current target layer.

19. The neural network device according to claim 15, wherein, Regarding the determination of the current target layer, the one or more processors are configured to: identify the layer, other than the previously pruned layers, that is predicted to have the lowest sensitivity to changes in the inference accuracy of the prediction as the current target layer.

20. The neural network device according to claim 15, wherein, For the prediction of changes in inference accuracy, the one or more processors are configured to: calculate the sensitivity of each of the plurality of layers based on the difference between the inference accuracy before pruning each of the plurality of layers and the inference accuracy after pruning each of the plurality of layers.

21. The neural network device according to claim 15, wherein, The layer predicted to have the lowest sensitivity corresponds to the layer predicted to have the least impact on the reduction of inference accuracy of the neural network.

22. The neural network device according to claim 15, wherein, The one or more processors are configured to set the weight value corresponding to the target pruning rate as a weight threshold when the determined weight distribution corresponds to a standard normal distribution.

23. The neural network device according to claim 15, wherein, The steps for updating the weight threshold include: increasing the current weight threshold when the inference accuracy of the neural network, which includes weights pruned to the current weight threshold, is not reduced to the threshold accuracy.

24. The neural network device according to claim 15, wherein, The one or more processors are configured to: repeatedly perform the determination of the current target layer and the pruning of the determined current target layer until the target pruning rate is reached, or the pruning of all layers of the neural network or a predetermined number of layers in all layers is completed.

25. The neural network device according to claim 15, wherein, The trimmed dataset includes one of the following: an image set generated by randomly extracting a predetermined number of images for each class included in a given image set, and an image set generated by selecting valid classes from the trimmed dataset and randomly extracting a predetermined number of images for each selected valid class.

26. The neural network device according to claim 25, wherein, The one or more processors are configured to provide a pruned dataset through the following steps: A predetermined number of images are randomly extracted from each class included in a given image set; By using a neural network to perform inference on randomly selected samples, a label corresponding to each sample in the randomly selected samples is determined; The classes included in a given image set with the proportion of their most frequent labels equal to or greater than a predetermined accuracy are identified as valid classes; and A predetermined number of images are randomly extracted from each of the defined valid classes.

27. The neural network device of claim 15, wherein the one or more processors are further configured to: acquire an image to be recognized; and The generated, pruned neural network is used to perform inference on the acquired image.

28. The neural network device according to claim 15, wherein, The neural network device is any one of a personal computer, server device, mobile device, smartphone, autonomous vehicle, robot device, tablet device, augmented reality device, and Internet of Things device.

Images