Neural network pruning system, neural network pruning method, and neural network pruning program

Abstract

A meta-learning system according to the present disclosure includes a sub-network generation unit, a parameter learning unit, and a meta-parameter learning unit. The sub-network generation unit generates, from a neural network, a plurality of sub-networks by using a mask indicating pruning of the neural network. The parameter learning unit trains the plurality of sub-networks by giving a plurality of different training datasets to the plurality of sub-networks, respectively. The meta-parameter learning unit updates the mask on the basis of a test which is for assessing the plurality of trained sub-networks and in which a plurality of different test datasets respectively corresponding to the plurality of different training datasets are used for the plurality of trained sub-networks.

Description

[Technical Field]

[0001] The present disclosure relates to a neural network pruning system, a neural network pruning method, and a neural network pruning program. [Background technology]

[0002] One method for training neural networks is to optimize the branches of the neural network rather than the weights of the neural network. Pruned neural networks have attracted attention as lightweight and fast models. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Vivek Ramanujan, 4 others, “What's Hidden in a Randomly Weighted Neural Network?”, arXiv, 1911.13299v2, 2020. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above prior art techniques may have difficulty in providing a pruned neural network that can be well adapted to multiple tasks.

[0005] Therefore, the present disclosure provides a neural network pruning system, a neural network pruning method, and a neural network pruning program that can provide a pruned neural network that can be well adapted to multiple tasks. [Means for solving the problem]

[0006] In one aspect of the present disclosure, a neural network pruning system includes a generating unit that generates a plurality of sub-networks from a neural network by using a mask that indicates pruning of the neural network; a training unit that trains the plurality of sub-networks by providing a plurality of different training data sets to the plurality of sub-networks, respectively; and an updating unit that updates the mask based on a test that evaluates the trained plurality of sub-networks using a plurality of different test data sets that respectively correspond to the plurality of different training data sets for the trained plurality of sub-networks. [Effects of the Invention]

[0007] The neural network pruning system can provide pruned neural networks that are well adapted to multiple tasks. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of an example environment for services related to meta-learning. [Figure 2A] FIG. 2A shows an overview of one meta-learning process according to the present disclosure. [Figure 2B] FIG. 2B shows an overview of one meta-learning process according to the present disclosure. [Figure 2C] FIG. 2C shows an overview of one meta-learning process according to the present disclosure. [Figure 2D] FIG. 2D shows an overview of one meta-learning process according to the present disclosure. [Figure 3] FIG. 3 is a block diagram of an example configuration of a meta-learning system according to the present disclosure. [Figure 4] FIG. 4 is a flowchart illustrating an example of a process for learning meta parameters. [Figure 5] FIG. 5 shows an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0009] Several embodiments of the present disclosure are described in the accompanying drawings and the following description, but the present invention is not limited to these embodiments. The various features of these embodiments can be combined in various ways, provided that these features are not mutually inconsistent. Like reference numerals refer to like elements.

[0010] The following explanation is divided into 10 sections: 1. Introduction 2. Environment for Meta-Learning Services 3. Overview of meta-learning process 4. Construction of the Meta-Learning System 5. Details of meta-learning process 5-1. Repeated randomization of weight values 5-2. Related methods 6. Meta-learning process flowchart 7.Effects 8. Hardware Configuration 9. Summary of embodiments 10. Addendum

[0011] 1. Introduction Meta-learning is a technique for training neural networks. Meta-learning is an approach to achieving small dataset learning for neural networks. Models obtained through meta-learning can learn unknown tasks with high accuracy from small amounts of data. Therefore, meta-learning is a promising technique for reducing the cost of collecting data.

[0012] On the other hand, recent research (Ramanujan et al., arXiv:1911.13299) has led to the discovery of a new property of neural networks. This research found that highly accurate classification models can be obtained by extracting subnetworks from neural networks whose parameters are randomly initialized. Even if the neural network parameters are randomly initialized, well-extracted subnetworks from the neural network show high accuracy in classification tasks.

[0013] However, in the methods used in the above studies, the extracted subnetworks depend on training data for a specific task, making it difficult to transfer the extracted subnetworks to another dataset (i.e., an unknown task).

[0014] To solve the above problems, the meta-learning system according to the present disclosure performs one or more meta-learning processes described below.

[0015] [2. Environment for Meta-Learning Services] First, the environment for services related to meta-learning will be described with reference to FIG.

[0016] Figure 1 is a block diagram of an example environment for meta-learning services, environment 1. As shown in Figure 1, environment 1 includes a meta-learning system 100, a network 200, and a user device 300. Meta-learning system 100 is an example of a neural network pruning system.

[0017] The meta-learning system 100 is a system that performs processing for performing meta-learning on neural networks. In this specification, such processing is referred to as a meta-learning process. An overview of one meta-learning process is provided in Section 3. Various meta-learning processes are then described in detail in Section 5.

[0018] The meta-learning system 100 includes one or more computers, such as one or more servers. An example configuration of the meta-learning system 100 is described in Section 4.

[0019] The network 200 is a network such as a local area network (LAN), a wide area network (WAN), the Internet, etc. The network 200 connects the meta-learning system 100 and the user device 300.

[0020] User device 300 is a computer such as a client device. User device 300 requests a model generated by the meta-learning process and code for the meta-learning process from meta-learning system 100. In response to the request, meta-learning system 100 provides the generated model and code to the user.

[0021] [3. Overview of Meta-Learning Processing] An overview of one meta-learning process will now be described with reference to Figures 2A, 2B, 2C, and 2D. Note that this overview is not intended to limit the invention or the embodiments described in the following sections.

[0022] 2A, 2B, 2C, and 2D collectively illustrate an overview of one meta-learning process according to the present disclosure, outline 10. Overview 10 conceptually illustrates one meta-learning process.

[0023] The meta-learning process of the present disclosure introduces the idea of ​​pruning optimization into meta-learning. The objective is to find a sparse network structure suitable for learning various tasks with a small amount of data. As described below, the meta-learning system 100 extracts subnetworks common to multiple tasks from a neural network.

[0024] Before describing Overview 10, some terms used in this specification will be summarized.

[0025] A neural network is a function that transforms an input vector x into an output vector. This transformation function is obtained by alternately composing matrix operations and nonlinear functions. The variables included in the matrix operations and nonlinear functions are collectively expressed as θ = θ i (i=1,...N). In this specification, the variable θ is called a parameter of the neural network. A parameter is a collection of real values ​​(N-dimensional vector).

[0026] A task is a pair of small training data D_train and test data D_test. Examples of tasks include classification of dog types, bird types, etc. For example, the small training data D_train is a set of pairs of dog images and labels indicating the dog types. In this case, the test data D_test is also a set of pairs of dog images and labels indicating the dog types. The dog images in the test data D_test do not overlap with the dog images in the small training data D_train.

[0027] A sparse network structure is a network structure in which the matrices that make up a neural network are sparse. The sparse network structure is represented by an N-dimensional 0 / 1 vector (see, for example, Equation 5 described below with reference to FIG. 4). The dimension of the 0 / 1 vector is the same as the parameter of the neural network. In this specification, the 0 / 1 vector is referred to as a mask.

[0028] We now describe scheme 10. One meta-learning process shown in scheme 10 includes four steps.

[0029] In step S1, the meta-learning system 100 acquires a neural network. The meta-learning system 100 randomly initializes the parameters of the neural network. The parameter θ1 is an initial parameter.

[0030] In step S2, the meta-learning system 100 trains the pruned neural network using the mask. As shown in FIG. 2B, the meta-learning system 100 generates two copies of the neural network and then applies the mask m to the two copies. The meta-learning system 100 provides training datasets to the two pruned neural networks, respectively. The first neural network is trained using the training data for task A, and the second neural network is trained using the training data for task B. The parameters of the first neural network are changed from parameter θ1 to parameter θ2, and the parameters of the second neural network are changed from parameter θ1 to parameter θ3.

[0031] In step S3, the meta-learning system 100 calculates the value of a loss function. This loss function evaluates the error between the output vector of the trained neural network and the test dataset. As shown in Figure 2C, the loss function L(cursive) A and the loss function L (cursive) B correspond to task A and task B, respectively.

[0032] In step S4, the meta-learning system 100 updates the mask m. Each component of the mask m is calculated by (1) the loss function L A and the loss function L B and (2) the loss function L A and the loss function L B The update of mask m is described in more detail below with reference to FIG.

[0033] As described above, the meta-learning system 100 obtains a sparse network structure by optimizing the mask m. In this optimization, a pruned copy neural network is trained on training data for an unknown task. A sparse network structure is then obtained using the copy neural network and test data. The sparse network structure can perform high-accuracy inference on the test data. In task learning and inference, parameters corresponding to the zero mask are ignored, and parameters corresponding to the one mask are involved in task learning and inference.

[0034] In other words, the mask learns which parameters are needed for learning and inference of unknown tasks. The meta-learning system 100 can reduce the number of parameters during learning to the number essentially required. As a result, the meta-learning system 100 enables high-accuracy learning even when the amount of training data is small.

[0035] [4. Construction of Meta-Learning System] Next, an example of the configuration of the meta-learning system 100 will be described with reference to FIG.

[0036] 3 is a block diagram of an example configuration of a meta-learning system 100 according to the present disclosure. As shown in FIG. 1, meta-learning system 100 includes a communication unit 110, a control unit 120, and a storage unit 130. Meta-learning system 100 may also include an input unit (e.g., a keyboard, a mouse) that accepts input from an administrator of meta-learning system 100. Meta-learning system 100 may also include an output unit (e.g., a liquid crystal display, an organic electroluminescence (EL) display) that displays information to the administrator.

[0037] (Communication unit 110) The communication unit 110 is implemented by a network device such as a network interface card (NIC). The communication unit 110 is connected to the network 200 by wire or wirelessly. The communication unit 110 can transmit and receive data to and from the user device 300 via the network 200.

[0038] (control unit 120) The control unit 120 is implemented by a data processing device and various programs stored in a storage device. The data processing device is, for example, a processor such as a central processing unit (CPU), a micro processing unit (MPU), or a general purpose graphic processing unit (GPGPU). The control unit 120 may be implemented as a controller for controlling multiple operations of the meta-learning system 100. For example, when one or more processors execute a program (multiple instructions) by using a random access memory (RAM) as a working area, the one or more processors perform multiple operations.

[0039] (Storage unit 130) The storage unit 130 is implemented by a semiconductor memory such as RAM or flash memory, a magnetic disk such as a hard disk, or an optical disk. The storage unit 130 can store various programs and various data. The storage unit 130 can store various data used in the meta-learning process.

[0040] As shown in FIG. 3, the control unit 120 includes an initialization unit 121, a task sampling unit 122, a network copying unit 123, a sub-network generation unit 124, a parameter learning unit 125, an evaluation unit 126, and a meta-parameter learning unit 127. The initialization unit 121 is an example of a first acquisition unit and a second acquisition unit. The sub-network generation unit 124 is an example of a generation unit. The parameter learning unit 125 is an example of a training unit. The meta-parameter learning unit 127 is an example of an update unit. Data processing performed by each unit will be described below.

[0041] (Initialization unit 121) The initialization unit 121 initializes various data used in the meta-learning process. The initialization unit 121 can accept various data used in the meta-learning process from a user. For example, the initialization unit 121 can accept an initial sparse ratio (described later) from the user.

[0042] (Task sampling unit 122) The task sampling unit 122 samples the tasks used in the meta-learning process.

[0043] (Network Copy Unit 123) The network copy unit 123 copies the neural network.

[0044] (Sub-network generation unit 124) The sub-network generator 124 generates a mask for pruning the neural network.

[0045] (Parameter learning unit 125) The parameter learning unit 125 learns the parameters of the sub-network.

[0046] (Evaluation Section 126) The evaluation unit 126 evaluates the learning of the sub-network.

[0047] (Metaparameter learning unit 127) The meta parameter learning unit 127 learns the meta parameters of the neural network.

[0048] [5. Details of Meta-Learning Processing] One meta-learning process was outlined above with reference to Figures 2A, 2B, 2C and 2D. A variety of meta-learning processes are described in detail in this section.

[0049] [5-1. Repeated randomization of weight values] It is known that the learning results obtained by pruning optimization tend to be less accurate than those obtained by existing parameter update methods. In contrast, the meta-learning system 100 uses iterative randomization of weight values ​​(pruning optimization and re-randomization) to ensure that the accuracy of the learning results obtained in this way is comparable to that of learning results obtained by existing learning methods.

[0050] [5-2. Related Methods] (Neural Architecture Search (NAS)) NAS is a method for optimizing the combination of neural network blocks. NAS optimizes the combination of blocks that make up a neural network (linear layers, convolutional layers, skip connections, etc.), while leaving each block as a dense matrix without sparsifying it.

[0051] The meta-learning according to the present disclosure is different in that it does not combine blocks, but rather sparsifies the matrices that are components of each block.

[0052] (Model-Agnostic Meta-Learning (MAML)) MAML is a meta-learning method that optimizes the initial value θ that is most suitable for small-data learning of various tasks. On the other hand, the meta-learning according to the present disclosure differs from MAML in that it leaves θ fixed as a random initial value and instead optimizes m, which has a sparse structure, making it possible to identify which parameters are essentially necessary.

[0053] Furthermore, the meta-learning method according to the present disclosure prevents rote memorization of the task set used in the meta-learning by not directly learning parameters. Therefore, when a new task that is significantly different from the original task is given, the meta-learning method according to the present disclosure has the effect of learning with higher accuracy than MAML.

[0054] (Sparsification method using L2 / L1 regularization) The sparsification method using L2 / L1 regularization is a method of learning the parameter θ while simultaneously imposing constraints to reduce its magnitude. On the other hand, the meta-learning method according to the present disclosure differs in that the parameter θ is not learned but is fixed at a random initial value, and instead a 0 / 1 variable vector m is meta-learned.

[0055] (Pruning optimization (Edge-popup algorithm)) The edge-popup algorithm trains a neural network f(x;θ) by fixing the parameter θ at a random initial value and optimizing m (see Ramanujan et al., arXiv:1911.13299).

[0056] The Edge-popup algorithm is similar to the meta-learning method disclosed herein in that it optimizes m by fixing θ through random initialization, but Edge-popup is not a meta-learning method. Therefore, the Edge-popup algorithm has problems in that it can only be applied to one task and cannot learn accurately when learning from a small amount of data.

[0057] In addition, in Edge-popup, the sparse parameter ratio is fixed as a hyperparameter, and it is not possible to learn the optimal sparse ratio. On the other hand, the sparse network structure obtained by meta-learning according to the present disclosure differs from Edge-popup in that it can learn with high accuracy even with a small amount of data and can automatically learn the sparse ratio.

[0058] [6. Meta-learning process flowchart] Next, a flowchart of an example of a meta-learning process will be described with reference to Figure 4. The example of the meta-learning process includes a process for learning meta-parameters. The process for learning meta-parameters is performed, for example, by the meta-learning system 100 of Figure 1.

[0059] FIG. 4 is a flowchart showing process P100, which is an example of a process for learning meta parameters.

[0060] The initialization unit 121 of the meta-learning system 100 initializes a parameter θ of a neural network model f(x;θ) (step S101). The parameter θ is an N-dimensional parameter. The neural network model f(x;θ) is a function that converts an input vector x into an output vector.

[0061] In step S101, the initialization unit 121 calculates the number of dimensions N of the parameters and the parameter distribution D param The initialization unit 121 receives N random values ​​θ i (i=1,…N) is a parameter distribution D param Random values ​​θ are sampled independently from i is an N-dimensional parameter and belongs to the N-dimensional real number space. The initialization unit 121 initializes the random value θ i The initial parameters of the neural network model are θ=θ i Returns the number of rows (i=1,…N).

[0062] Parameter distribution D param is a distribution commonly used to initialize parameters in neural networks. For example, the parameter distribution D param is a distribution such as normal or uniform.

[0063] The initialization unit 121 initializes a meta parameter s (step S102). The meta parameter s is a parameter learned through meta-learning in a neural network model. Like the parameter θ, the meta parameter s is an N-dimensional parameter.

[0064] In step S102, the initialization unit 121 generates N random values ​​s i (i=1,…N) is a parameter distribution D param The initialization unit 121 samples N random values ​​s i The initial meta-parameter s of meta-learning is s=s i Returns the number of rows (i=1,…N).

[0065] The initialization unit 121 initializes a threshold t (step S103). The threshold t is a value used to find a sparse neural network structure in the neural network model. The threshold t is a real number. As will be described below, the threshold t is set to the value of each component (s i ) is the threshold for

[0066] In step S103, the initialization unit 121 receives an initial sparsity rate r. The initialization unit 121 calculates a threshold value t using the meta parameter s and the initial sparsity rate r. The initialization unit 121 calculates the threshold value t as follows:

number

number

number

[0067] The initial sparseness ratio r is a hyperparameter in the meta-learning process. A user of the meta-learning system 100 can freely set the initial sparseness ratio r. As explained below, a sparse neural network structure is obtained by this process P100. The initial sparseness ratio r controls the sparsity of this neural network structure. The smaller the initial sparseness ratio r, the denser the neural network structure. The larger the initial sparseness ratio r, the sparser the neural network structure.

[0068] The task sampling unit 122 of the meta-learning system 100 randomly samples tasks (D_train, D_test) from a meta-learning dataset (step S104). The meta-learning dataset is prepared in advance. The meta-learning dataset is a collection of tasks. A task (D_train, D_test) is a pair of a small amount of training data D_train and test data D_test.

[0069] The meta-learning dataset used by the task sampling unit 122 is similar to the dataset for meta-learning such as MAML. The small amount of training data D_train is a small amount of labeled data. The small amount of training data D_train is a collection of pairs of input data and labels, and the number of pairs is small.

[0070] The test data D_test is a small amount of labeled data similar to the small amount of training data D_train. For example, if D_test is data on the classification of dogs, the test data D_test is also data on the classification of dogs. Also, if D_test is data on the classification of birds, the test data D_test is also data on the classification of birds.

[0071] The network copying unit 123 of the meta-learning system 100 copies the neural network model f(x;θ) (step S105). As explained above, the copy of the neural network model is called a copy model.

[0072] In step S104, the network copy unit 123 copies each component of the parameter θ of the neural network model. The copy of the parameter θ is denoted by the parameter θ'. The network copy unit 123 generates a copy model f(x;θ'). The copy model f(x;θ') behaves in the same way as the original neural network model f(x;θ). Changes to the parameter θ' are not reflected in the parameter θ.

[0073] The sub-network generation unit 124 of the meta-learning system 100 generates a sub-network (Equation 4 below) of the copy model f(x;θ′) (step S106).

number

[0074] Each component of the variable vector m is 0 or 1. The variable vector m is defined as follows (Equation 5 below).

number

[0075] The sub-network generation unit 124 calculates the sub-networks of each component (m i ) for each component of the meta parameter s (s i ) and threshold t. If s i If ≦t, then m i =0. If t i If so, m i ​= 1. In this way, the sub-network generation unit 124 calculates each component of the variable vector m based on whether the corresponding component of the meta parameter s is greater than the threshold value t.

[0076] The parameter learning unit 125 of the meta-learning system 100 trains the parameter θ′ of the sub-network f(x;θ′ \odot m) using the small amount of training data D_train (step S107). After the training, the parameter of the sub-network f changes from θ′ to θ″.

[0077] In addition, component m i θ corresponding to =0 i ' does not appear in the computation graph representing the sub-network f. Therefore, the parameter learning unit 125 i θ corresponding to =1 i Only ´ is trained using the small amount of training data D_train.

[0078] After the learning, the evaluation unit 126 of the meta-learning system 100 evaluates the accuracy of the copy model f(x;θ′) using the test data D_test (step S108).

[0079] Specifically, the accuracy is the value of a loss function that evaluates the error between the output vector of the sub-network f and the test data D_test. The evaluation unit 126 calculates the value of the loss function for the test data D_test, and then returns the calculation result and the corresponding computation graph.

[0080] As will be explained below, the computation graph is used to learn the meta-parameter s. Therefore, the meta-learning system 100 maintains the computation graph used in steps S105 and S106.

[0081] The meta parameter learning unit 127 of the meta-learning system 100 applies error backpropagation to the stored computation graph, and then updates the meta parameter s in a direction that improves the accuracy for the test data D_test (step S109).

[0082] In step S109, the meta parameter learning unit 127 applies backpropagation to the computation graph and then calculates a gradient vector g with respect to the variable vector m. This gradient vector g is the gradient of the variable vector m with respect to the loss function. The meta parameter learning unit 127 updates the meta parameter s using the gradient vector g and the value of the loss function for the test data D_test. The meta parameter learning unit 127 updates the meta parameter s as follows:

number

[0083] Note that the variables in the variable vector m are discrete variables. Therefore, the variable vector m is a non-differentiable function. However, backpropagation can be applied to the meta parameter s, and the meta parameter learning unit 127 can calculate the gradient vector g when the variables in the variable vector m are considered to be continuous variables. This calculation method is called a straight-through estimator. The meta parameter learning unit 127 can update the meta parameter s using the straight-through estimator.

[0084] The meta-learning system 100 discards the copy model f(x;θ′) and the computation graph (step S110).

[0085] The meta-learning system 100 determines whether the number of times the processes from step S104 to step S109 have been executed is equal to or greater than a specified value (step S111). If it is determined that the number of times the processes have been executed is not equal to or greater than the specified value (step S111: No), the meta-learning system 100 performs step S104 again. If it is determined that the number of times the processes have been executed is equal to or greater than the specified value (step S111: Yes), process P100 ends.

[0086] [7. Effects] The meta-learning system 100 can find a task-specific sparse network structure suitable for learning various tasks using a small amount of data. This allows the meta-learning system 100 to automatically acquire a sparse structure suitable for learning, such as a convolutional neural network (CNN) for image classification, from meta-learning data.

[0087] [8. Hardware Configuration] 5 is a diagram showing an example of a computer hardware configuration, ie, a computer 1000. The systems and methods described in this specification are implemented by the computer 1000, for example.

[0088] Computer 1000 is an example of a computer that implements meta-learning system 100 by executing a program. Computer 1000 includes memory 1010 and a CPU 1020. Computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0089] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium (for example, a magnetic disk or an optical disk) can be inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0090] The hard disk drive 1090 stores an OS 1091, an application program 1092, a program module 1093, and program data 1094. The programs executed by the computer 1000 define multiple operations of the meta-learning system 100. These programs may be implemented as program modules 1093 written in code executable by the computer 1000. The program modules 1093 are stored, for example, on the hard disk drive 1090. For example, the hard disk drive 1090 stores program modules 1093 for executing processes similar to the functions of the components of the meta-learning system 100. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0091] The hard disk drive 1090 can store a neural network pruning program for the meta-learning process. The hard disk drive 1090 may store a computer program product including the neural network pruning program (a plurality of instructions). When executed, the neural network pruning program performs one or more of the methods described above.

[0092] The setting data used in the various processes described above may be implemented as program data 1094. The setting data is stored in, for example, the memory 1010 or the hard disk drive 1090. The CPU 1020 loads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as needed. Then, the CPU 1020 performs the various processes described above.

[0093] The program module 1093 and the program data 1094 may be stored in a removable storage medium instead of the hard disk drive 1090. The CPU 1020 may load the program module 1093 and the program data 1094 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected to the computer 1000 via a network (such as a LAN or WAN). In this case, the CPU 1020 may load the program module 1093 and the program data 1094 via the network interface 1070.

[0094] 9. Summary of Embodiments As described above, meta-learning system 100 includes subnetwork generator 124, parameter learner 125, and meta-parameter learner 127. In at least one embodiment, subnetwork generator 124 generates multiple subnetworks from a neural network by using a mask that indicates neural network pruning. In at least one embodiment, parameter learner 125 trains the multiple subnetworks by providing multiple different training data sets to the multiple subnetworks. In at least one embodiment, meta-parameter learner 127 updates the mask based on a test that evaluates the trained multiple subnetworks using multiple different test data sets that correspond to the multiple different training data sets for each of the trained multiple subnetworks.

[0095] In some embodiments, the sub-network generator 124 generates multiple copies of the neural network based on multiple parameters of the neural network, and generates multiple sub-networks by applying a mask to the multiple copies of the neural network.

[0096] As described above, meta-learning system 100 includes an initialization unit 121. In at least one embodiment, initialization unit 121 obtains a plurality of score parameters indicative of the evaluation of a plurality of branches in the neural network. In some embodiments, initialization unit 121 obtains a sparseness fraction that controls the sparsity of the neural network. In some embodiments, subnetwork generator 124 generates a mask indicative of pruning of the neural network based on the plurality of score parameters and the sparseness fraction.

[0097] In some embodiments, the initialization unit 121 randomly samples the score parameters from a predetermined parameter distribution.

[0098] In some embodiments, the initialization unit 121 accepts the sparse ratio from a user.

[0099] In some embodiments, the meta parameter learner 127 updates score parameters based on tests evaluating the trained sub-networks, and generates a new mask indicating pruning of the neural network based on the updated score parameters and the sparse fraction.

[0100] [10. Addendum] Finally, the above description is supplemented with other embodiments. Various embodiments have been described above with reference to the drawings. These embodiments are exemplary, and the above description is not intended to limit the present disclosure to these embodiments. The features described in this specification can be implemented in various ways, including modifications and improvements based on the knowledge of those skilled in the art.

[0101] (various variations) In this specification, some processes have been described as being performed automatically. Some of these processes may be performed manually. Some other processes have been described as being performed manually. All or part of these other processes may be performed automatically using known methods.

[0102] Various implementations of the meta-learning system 100 are described herein or illustrated in the drawings. Some implementations relate to information including various data, data processing procedures, specific names, or parameters. Such implementations may be modified in any way unless otherwise specified. For example, the various data are not limited to the data illustrated in the drawings.

[0103] Components of the system are shown in the drawings. The illustrated components conceptually illustrate the functions of the system. The components are not necessarily physically configured as shown in the drawings. The components may be integrated or distributed, and the specific form of the system is not limited to the illustrated form. All or part of the system may be functionally or physically integrated or distributed depending on various loads and usage conditions.

[0104] (Terms that represent components) The term "module," "section," "-er suffix," or "-or suffix" can be read as "unit," "means," "circuit," etc. For example, a communication module, a control module, and a storage module can be read as a communication unit, a control unit, and a storage unit, respectively.

[0105] (Configuration of control unit) The configuration of the control unit 120 shown in Fig. 3 is exemplary, and the data processing described with respect to a particular unit may not necessarily be performed by that particular unit. For example, the meta parameter learning unit 127 may perform the data processing described with respect to the evaluation unit 126. Furthermore, the control unit 120 may include other units not shown in Fig. 3. The other units may perform the data processing described with respect to the control unit 120.

[0106] (Data Processing Device) The data processing device described for the control unit 120 is not limited to the specific hardware described above, and may be, for example, various types of computers or integrated circuits such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a GPGPU (General Purpose Graphic Processing Unit). [Explanation of symbols]

[0107] 1 Environment 100 Meta-Learning Systems 110 Communications Department 120 control section 121 Initialization section 122 Task Sampling Unit 123 Network Copy Department 124 Subnetwork generation unit 125 Parameter Learning Unit 126 Evaluation Department 127 Metaparameter Learning Unit 130 Storage section 200 Network 300 User Device

Claims

1. a generator for generating a plurality of sub-networks from a neural network by using a mask indicating pruning of the neural network; a training unit for training the plurality of sub-networks by providing a plurality of different training data sets to the plurality of sub-networks, respectively; an update unit that updates the mask based on a test that evaluates the plurality of trained sub-networks using a plurality of different test data sets that respectively correspond to the plurality of different training data sets for the plurality of trained sub-networks; A neural network pruning system comprising:

2. The generator generates multiple copies of the neural network based on multiple parameters of the neural network, and generates multiple sub-networks by applying the mask to the multiple copies of the neural network.

2. The neural network pruning system of claim 1.

3. a first acquisition unit that acquires a plurality of score parameters that indicate evaluations of a plurality of branches in the neural network; a second acquisition unit that acquires a sparse ratio that controls the sparsity of the neural network; Furthermore, The generator generates the mask indicating pruning of the neural network based on the score parameters and the sparse ratio.

2. The neural network pruning system of claim 1.

4. The first acquisition unit randomly samples the plurality of score parameters from a predetermined parameter distribution.

4. The neural network pruning system of claim 3.

5. The second acquisition unit receives the sparse ratio from a user.

4. The neural network pruning system of claim 3.

6. The update unit updates the score parameters based on the test evaluating the trained sub-networks, and generates a new mask indicating pruning of the neural network based on the updated score parameters and the sparse fraction.

4. The neural network pruning system of claim 3.

7. 1. A computer-implemented method for neural network pruning, comprising: generating a plurality of sub-networks from a neural network by using a mask indicating pruning of said neural network; training the plurality of sub-networks by respectively providing a plurality of different training data sets to the plurality of sub-networks; updating the mask based on a test that evaluates the plurality of trained sub-networks using a plurality of different test data sets that respectively correspond to the plurality of different training data sets for the plurality of trained sub-networks; A neural network pruning method including:

8. A neural network pruning program for causing a computer to function as the neural network pruning system according to any one of claims 1 to 6.

Images