Information processing device, information processing method, and program

Abstract

To further accurately execute learning by distillation of a model reduced in size.SOLUTION: An information processing device includes a target model learning unit, a change unit, a selection unit, and a student model learning unit. The target model learning unit learns a target model to be reduced in size. The change unit changes the target model to a student model having a smaller size than the target model. The selection unit selects, as a teacher model, any one of a plurality of models including the target model and one or more intermediate models smaller in size than the target model in accordance with comparison results between the size of the target model and the size of the student model. The student model learning unit learns the student model by distillation using the selected teacher model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In neural networks such as deep neural networks (DNNs), in consideration of high-speed model learning in environments with limited computing resources such as edge devices, a process of reducing the size of the model may be performed to reduce the amount of computation and memory capacity of the model. In addition, in order to improve the accuracy of the reduced-size model, a technique is known in which a student model, which is a reduced-size model, is learned by distillation (also called knowledge distillation) using the model before reduction as a teacher model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-509892 [Patent Document 2] Patent Publication No. 2022-056412 [Patent Document 3] Special Publication No. 2022-549806 [Non-patent literature]

[0004] [Non-Patent Document 1] Hao Li, et al., “PRUNING FILTERS FOR EFFICIENT CONVNETS”, arXiv:1608.08710v3 10 Mar 2017 [Non-Patent Document 2] A. Yaguchi, et al., “Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks”, Proc. of 17th IEEE International Conference on Machine Learning and Applications (ICMLA) 2018. [Non-Patent Document 3] Tao Huang, et al., “Knowledge Distillation from A Stronger Teacher”, arXiv:2205.10536v3 28 Dec 2022 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide an information processing device, an information processing method, and a program that are capable of executing learning by distillation of a reduced-size model with higher accuracy. [Means for solving the problem]

[0006] An information processing device according to an embodiment includes a target model learning unit, a change unit, a selection unit, and a student model learning unit. The target model learning unit learns a target model to be reduced in size. The change unit changes the target model to a student model smaller in size than the target model. The selection unit selects, as a teacher model, one of a plurality of models including the target model and one or more intermediate models smaller in size than the target model, according to a comparison result between the size of the target model and the size of the student model. The student model learning unit learns the student model by distillation using the selected teacher model. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of changing the target model to a student model. [Diagram 3] FIG. 13 is a diagram showing an example of generating an intermediate model and a student model from a target model. [Figure 4] FIG. [Diagram 5] FIG. 1 shows an example of a student model. [Figure 6] FIG. [Figure 7] FIG. 4 is a diagram for explaining an outline of a selection process of initial parameters. [Figure 8] FIG. 11 is a diagram showing an example of a process for determining a combination. [Figure 9] FIG. 11 is a diagram showing a schematic diagram of selection of a combination using an evaluation value. [Figure 10] FIG. 11 is a diagram showing a schematic diagram of selection of a combination using an evaluation value. [Figure 11] FIG. 11 is a diagram showing a schematic diagram of selection of a combination using an evaluation value. [Figure 12] 4 is a flowchart of a learning process in the embodiment. [Figure 13] FIG. 13 is a block diagram of an information processing apparatus according to a modified example. [Figure 14] FIG. 2 is a hardware configuration diagram of the information processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of an information processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

[0009] In the process of reducing the size of a model, the ratio of the size of the student model to the size of the teacher model (size reduction rate) may vary. For example, if the reduction rate is large, the accuracy of learning by distillation may decrease due to the large difference between the student model and the teacher model.

[0010] In the following embodiment, a teacher model is selected according to the size of the student model (model after size reduction) to be learned by distillation. In addition, the initial values ​​(initial parameters) of the parameters during learning of the student model are selected from the parameters of the model to be reduced in size (hereinafter, the target model). This makes it possible to execute learning by distillation of the reduced-size model with higher accuracy.

[0011] In addition, in the conventional technology, if pre-training is required for the target model, the student model after size reduction also needs to be pre-trained. In the embodiment, by appropriately selecting the initial parameters, pre-training of the student model can be eliminated, thereby reducing the cost of training.

[0012] In an embodiment, one of a plurality of models including a target model and one or more intermediate models is selected as a teacher model according to a ratio of sizes of the student model and the target model. The intermediate model is a model smaller in size than the target model. In addition, an initial parameter of the student model is selected based on an evaluation value from the initial parameter of the target model.

[0013] In the following, a case where the models (target model, intermediate model, student model) are neural network models (hereinafter, simply referred to as neural networks) will be described as an example. The models are not limited to this, and may be models of any other structure. A neural network includes multiple layers, each of which includes multiple elements.

[0014] For example, when the neural network is a convolutional neural network (CNN), the multiple elements may be multiple channels included in a convolution layer. The elements are not limited to this, and may be any other elements. For example, the elements may be nodes included in each layer of the neural network. In the following, an example will be described in which the elements are channels.

[0015] Fig. 1 is a block diagram showing an example of a configuration of an information processing device 100 according to an embodiment. As shown in Fig. 1, the information processing device 100 includes a receiving unit 101, a target model learning unit 102, a student model learning unit 103, an output control unit 104, a change unit 110, a selection unit 120, and a storage unit 131.

[0016] The receiving unit 101 receives input of various information used in the information processing device 100. For example, the receiving unit 101 receives information indicating a target model (for example, parameters such as weights and biases), a plurality of learning data (learning data sets) used for learning the model, and the like.

[0017] The object model training unit 102 trains an object model to be reduced in size. For example, the object model training unit 102 trains the object model using a training data set for training.

[0018] The modification unit 110 modifies the target model into a student model that is smaller in size than the target model. The student model is a model that is learned by distillation using a teacher model. For example, the modification unit 110 modifies the target model into the student model by deleting a part of the channels included in each of a plurality of layers included in the target model. Note that modifying the target model into the student model can also be interpreted as being equivalent to generating a student model from the target model.

[0019] Any method may be used for modifying the student model by the modifying unit 110, but for example, the following methods (M1) to (M3) can be applied. Note that in each of the methods shown below, one or more intermediate models can be generated along with the student model.

[0020] (M1) A student model is generated by uniformly reducing the number of channels (number of channels) in each of multiple layers included in the target model. For example, the number of channels in each layer of the target model is reduced by a specified ratio (hereinafter also referred to as scale). Hereinafter, changing (increasing or decreasing) the number of channels by a specified scale may be referred to as scaling.

[0021] The channels to be reduced by scaling may be selected in any manner, including random selection, continuous selection of a number of channels according to the scale from a specific position (the beginning, middle, end, etc.), and the like.

[0022] FIG. 2 is a diagram showing an example in which a target model 201 is changed to a student model 202 by (M1). The target model 201 includes four layers L1 to L4. The student model 202 includes four layers L1-2 to L4-2. The numerical values ​​written under each layer indicate the number of channels included in the corresponding layer. FIG. 2 shows an example in which 0.5 is specified as the scale. As shown in FIG. 2, the number of channels in each of the four layers included in the target model 201 is reduced according to the scale of 0.5, and the student model 202 is generated.

[0023] In addition, the modification unit 110 may modify the target model into a plurality of models according to a plurality of scales, and one of the modified models may be the student model, and the remaining may be the intermediate model. For example, the modification unit 110 may scale the target model into the intermediate model using a scale of 0.75, and scale the target model into the student model using a scale of 0.5. In this manner, the intermediate model may be a model larger in size than the student model. The intermediate model may be a model of the same size as the student model, or a model smaller in size than the student model.

[0024] (M2) A student model is generated by pruning the target model. Pruning is, for example, a process of finding operations that can be reduced among multiple operations realized by the target model, and generating a model in which the operations have been reduced. Any method of pruning may be used, but the methods of Non-Patent Document 1 and Non-Patent Document 2, for example, can be applied.

[0025] The modification unit 110 may generate models of multiple sizes by changing the pruning strength, and use one of the generated models as a student model and the remaining as an intermediate model. For example, in a method of calculating the importance for each channel as in Non-Patent Document 1, the strength can be manipulated by changing a threshold (threshold of importance) for determining which channel to reduce. In a method using compaction to learn a sparse model in which many 0s are included in the weights as in Non-Patent Document 2, the strength can be manipulated by changing a parameter (e.g., weight decay) during learning.

[0026] FIG. 3 is a diagram showing an example of generating intermediate model 301 and student model 302 from target model 201 by (M2). Intermediate model 301 includes four layers L1-3 to L4-3. Student model 302 includes four layers L1-4 to L4-4. Intermediate model 301 corresponds to a model generated at low intensity. Student model 302 corresponds to a model generated at higher intensity than intermediate model 301. Because the intensity is relatively low, the degree of deterioration in accuracy of intermediate model 301 is relatively smaller than that of student model 302.

[0027] (M3) Generate one or more intermediate models by pruning the target model, and generate a student model by scaling one of the generated intermediate models.

[0028] 4 is a block diagram showing an example of the configuration of the modification unit 110 when (M3) is applied. As shown in FIG.

[0029] The pruning unit 111 generates one or more intermediate models by pruning the target model. The generation unit 112 generates a student model by scaling the size of each of a plurality of layers included in one of the intermediate models by a specified scale.

[0030] FIG. 5 is a diagram showing an example of a student model 501 generated by (M3). The student model 501 includes four layers L1-5 to L4-5. The numbers written at the top of each layer indicate the number of channels included in the corresponding layer before and after pruning. The numbers to the right of the symbol " / " indicate the number of channels before pruning, and the numbers to the left of the symbol " / " indicate the number of channels after pruning. The numbers written at the bottom of each layer indicate the number of channels included in the corresponding layer after scaling. In this example, the scale is 0.5.

[0031] 1, the selection unit 120 selects a teacher model to be used in learning by distilling a student model. The selection unit 120 also selects initial values ​​(initial parameters) of parameters during learning of the student model. The parameters are, for example, weights and biases.

[0032] 6 is a block diagram showing a detailed functional configuration example of the selection unit 120. As shown in FIG. 6, the selection unit 120 includes a teacher model selection unit 121 and an initial parameter selection unit 122.

[0033] The teacher model selection unit 121 selects one of a plurality of models including the target model and one or more intermediate models as a teacher model, for example, according to a result of comparing the size of the target model with the size of the student model. For example, the teacher model selection unit 121 calculates the ratio of the size of the student model to the size of the target model (= size of the student model / size of the target model). The size may be calculated in any way, but is calculated, for example, from the size of the model parameters (data size) and the number of operations (FLOPs) during inference using the model.

[0034] The teacher model selection unit 121 compares the calculated ratio with a threshold, and selects the target model as the teacher model if the ratio is greater than the threshold, and selects the intermediate model as the teacher model if the ratio is equal to or less than the threshold. When two or more intermediate models are generated, the teacher model selection unit 121 may use multiple thresholds to select an intermediate model of a size according to the magnitude of the ratio. The threshold is, for example, predetermined according to the type of the target model.

[0035] The initial parameter selection unit 122 selects initial parameters (initial values) of a plurality of layers included in the student model from the parameters of a plurality of layers included in the target model.

[0036] The initial parameters are, for example, initial values ​​of parameters to be set for one or more channels to be included in each layer at the start of learning. For example, the initial parameter selection unit 122 selects initial parameters for one or more channels to be included in each of the multiple layers included in the student model from parameters of multiple channels included in the multiple layers included in the target model.

[0037] The parameters of the multiple channels included in the multiple layers included in the target model may be parameters before the target model is trained (initial values ​​of the parameters) or may be parameters after the target model is trained.

[0038] As a method for selecting the initial parameters, for example, the following method can be applied. First, for each of a plurality of layers (first layer) included in the intermediate model, the initial parameter selection unit 122 determines a plurality of combinations (a plurality of element groups) including some channels selected from a plurality of channels included in the layer. From among the plurality of combinations, the initial parameter selection unit 122 selects a channel included in a combination having a larger evaluation value than other combinations. For the selected channel, the initial parameter selection unit 122 selects, as the initial parameter, a parameter set for the corresponding channel of the corresponding layer of the target model.

[0039] Fig. 7 is a diagram for explaining an outline of the selection process of the initial parameters. Fig. 7 shows an example of selecting initial parameters for layer L3 among four layers L1 to L4 included in the student model. The same process is performed for the other layers L1, L2, and L4.

[0040] 7 also shows an example in which the number of channels in layer L3 of the target model 701 is 8, and an intermediate model 702 in which the number of channels in layer L3 is 6 is generated by pruning. The initial parameter selection unit 122 finds multiple combinations including three channels from the six channels included in layer L3 of the intermediate model 702.

[0041] 8 is a diagram showing an example of finding three combinations. The numerical values ​​in the intermediate model 702 are information for identifying the channels. The initial parameter selection unit 122 selects three mutually different channels from the six channels of the intermediate model 702 to find three combinations 703-1, 703-2, and 703-3. Note that the number of combinations is not limited to three, and may be two or four or more.

[0042] Returning to FIG. 7, the initial parameter selection unit 122 selects three channels included in the combination with the large evaluation value among the three combinations. A model including the selected three channels corresponds to the student model 703. The initial parameter selection unit 122 selects and sets the initial parameters of each channel included in the layer L3 of the student model 703 from the parameters of the corresponding channel of the layer L3 of the target model 701. The parameters selected at this time may be the parameters (initial values ​​of the parameters) of the target model 701 before learning, or the parameters after learning the target model 701. The initial parameter selection unit 122 may select and set the initial parameters from the parameters of the corresponding channel of the layer L3 of the intermediate model 702 after pruning.

[0043] The evaluation value is a value for evaluating whether the channels included in each combination are appropriate, and for example, the following indices (E1) to (E3) can be used. In the following, a student model in which the parameters of the channels of the target model corresponding to the channels included in the combination are set as initial values ​​is referred to as a candidate model. An evaluation value is calculated for each candidate model. (E1) A value that represents the accuracy of inference using the candidate model (such as the accuracy rate of inference) (E2) The smaller the difference between the output statistics of the layer included in the candidate model and the output statistics of the corresponding layer included in the intermediate model when inference using the candidate model is performed on multiple data, the larger the value. (E3) A value that increases as the difference between the gradient statistics calculated for a layer included in the candidate model and the gradient statistics calculated for the corresponding layer included in the intermediate model becomes smaller when backward processing using the candidate model is performed on multiple data.

[0044] The statistics used in (E2) and (E3) may be any values, for example, the average, variance, maximum, and minimum. A value obtained by combining two or more statistics may be used as the evaluation value. For example, the sum of absolute values ​​or the sum of squares of the differences of multiple statistics may be used as the evaluation value. The backward processing performed in (E3) is a process of calculating the gradient (differential) of each layer with respect to the error of the model in the direction from the output layer of the model to the input layer.

[0045] 9 to 11 are diagrams each showing a schematic diagram of the selection of a combination (student model) using the evaluation values ​​of (E1) to (E3), respectively.

[0046] 9 shows an example of evaluation values ​​(accuracies) of each combination 703-1, 703-2, and 703-3. For example, the initial parameter selection unit 122 learns a candidate model corresponding to each combination using learning data. The number of times of learning may be, for example, a number that allows the accuracy of the candidate models after learning to be compared. The initial parameter selection unit 122 executes inference for each candidate model after learning, and calculates the accuracy of the inference (such as accuracy rate) as an evaluation value. In the example of FIG. 9, the initial parameter selection unit 122 selects the combination 703-3 corresponding to the candidate model with the largest evaluation value.

[0047] 10, the initial parameter selection unit 122 executes inference for each candidate model using a plurality of data. For each layer, the initial parameter selection unit 122 compares the statistics of the output of the layer when inference is executed for the plurality of data with the statistics of the output of the corresponding layer of the intermediate model. The initial parameter selection unit 122 selects the student model (candidate model) with the closest statistics (smallest difference).

[0048] In Fig. 11, the initial parameter selection unit 122 performs inference and backward processing for each candidate model using multiple data. For each layer, the initial parameter selection unit 122 compares the gradient statistics calculated by the backward processing for the multiple data with the gradient statistics calculated for the corresponding layer of the intermediate model. The initial parameter selection unit 122 selects the student model (candidate model) with the closest statistics (smallest difference).

[0049] The initial parameter selection unit 122 selects initial parameters for each layer of the student model corresponding to the combination selected as shown in FIGS. 9 to 11 from the parameters of the corresponding layer included in the target model.

[0050] Returning to Fig. 1, student model learning unit 103 learns a student model by distillation using the teacher model selected by teacher model selection unit 121. At this time, student model learning unit 103 learns the student model using initial parameters selected by initial parameter selection unit 122.

[0051] The output control unit 104 controls the output of various information used in the information processing device 100. For example, the output control unit 104 outputs information (parameters, etc.) indicating a trained student model to an external device that executes inference using the student model.

[0052] Each of the above units (reception unit 101, target model learning unit 102, student model learning unit 103, output control unit 104, change unit 110, and selection unit 120) is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by using a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0053] The storage unit 131 stores various information used in the information processing device 100. For example, the storage unit 131 stores information accepted by the acceptance unit 101 (information indicating a target model, learning data, etc.), information indicating a learned student model, and the like.

[0054] The storage unit 131 can be configured from any commonly used storage medium, such as a flash memory, a memory card, a RAM (Random Access Memory), a HDD (Hard Disk Drive), or an optical disk.

[0055] The information processing device 100 may be physically configured by one device, or may be physically configured by multiple devices. For example, the information processing device 100 may be constructed on a cloud environment. Furthermore, each unit in the information processing device 100 may be provided in a distributed manner in multiple devices.

[0056] Next, a learning process by the information processing device 100 of the embodiment will be described. The learning process is a process in which a student model, which is a reduced size of a target model, is learned by distillation using a teacher model. Fig. 12 is a flowchart showing an example of the learning process in the embodiment.

[0057] The object model learning unit 102 learns an object model using a learning data set (step S101). The modification unit 110 changes the size of the learned object model to generate a student model (step S102).

[0058] The selection unit 120 (teacher model selection unit 121) selects either the target model or the intermediate model as a teacher model to be used for learning the student model (step S103). For example, the teacher model selection unit 121 calculates the ratio of the student model size to the target model size, and selects the target model as the teacher model if the ratio is greater than a threshold. If the ratio is equal to or less than the threshold, the teacher model selection unit 121 selects the intermediate model as the teacher model.

[0059] The selection unit 120 (initial parameter selection unit 122) selects initial values ​​(initial parameters) of parameters used in learning of the student model (step S104).

[0060] The student model learning unit 103 uses the selected initial parameters to learn a student model by distillation using the selected teacher model (step S105).

[0061] The output control unit 104 outputs information (parameters, etc.) indicating the trained student model (step S106), and ends the learning process.

[0062] In the above description, the selection unit 120 executes both the selection of a teacher model (teacher model selection unit 121) and the selection of initial parameters (initial parameter selection unit 122). The information processing device 100 may be configured to execute only one of these.

[0063] If the system is provided with a function for selecting a teacher model taking into account the size of the student model, for example, when the difference between the student model and the target model is large, an intermediate model with a smaller difference can be selected as the teacher model, thereby suppressing the deterioration of learning accuracy caused by the difference between the student model and the teacher model.

[0064] Furthermore, if a function for selecting the initial parameters of the student model from the parameters of the target model is provided, more appropriate initial parameters can be set compared to a method in which the initial parameters are initialized with random numbers, for example. This makes it possible to execute learning of the student model with higher accuracy.

[0065] In this manner, the information processing device of the embodiment can perform learning by distillation of a reduced-sized model with higher accuracy.

[0066] (Modification) The information processing device may further include a function of pre-learning a target model. In the modified example, an example having a pre-learning function will be described. FIG. 13 is a block diagram showing an example of a configuration of an information processing device 100-2 of the modified example. As shown in FIG. 13, the information processing device 100-2 includes a receiving unit 101, a target model learning unit 102-2, a student model learning unit 103, an output control unit 104, a change unit 110, a selection unit 120, and a storage unit 131.

[0067] In the modified example, the function of the object model learning unit 102-2 is different from that of the above embodiment. The other configurations and functions are the same as those of the information processing device 100 of the above embodiment in FIG. 1, so the same reference numerals are used and the description thereof is omitted here.

[0068] The target model learning unit 102-2 differs from the target model learning unit 102 of the above embodiment in that it further executes pre-learning of the target model. For example, the target model learning unit 102-2 pre-learns the target model using a learning dataset DS_A (first dataset) for pre-learning, and learns the pre-learned target model using a learning dataset DS_B (second dataset) different from the learning dataset DS_A.

[0069] The learning dataset DS_A is, for example, a large-scale dataset prepared in advance that can learn general features and can be easily transferred to other tasks. The learning dataset DS_B is, for example, a dataset for training to become a model more suitable for a specific task.

[0070] When pre-learning is performed, the initial parameter selection unit 122 may select the parameters of the target model after pre-learning as the initial parameters of the student model. This makes it possible to eliminate the need for pre-learning of the student model, even when a target model that requires pre-learning is used. This makes it possible to reduce the cost of learning.

[0071] The information processing device of the above embodiment and modified example can be applied to learning of models used in various systems. The system may be any system that includes a function using a model, but may also be a system that uses a model on an edge terminal with a small storage capacity, for example, as follows. A detection system in which a server device detects objects such as people (people flow) using information detected by an edge terminal (for example, information detected from images obtained by an imaging device (camera)). A control system in which a control device controls the equipment to be controlled (lighting equipment, elevators, etc.) using information detected by edge devices (for example, information detected from sensor data obtained from sensors).

[0072] Next, a hardware configuration of the information processing apparatus according to the embodiment will be described with reference to Fig. 14. Fig. 14 is an explanatory diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment.

[0073] The information processing device of the embodiment includes a control device such as a CPU 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM 53, a communication I / F 54 that connects to a network and performs communication, and a bus 61 that connects each unit.

[0074] The programs executed by the information processing apparatus according to the embodiment are provided in a state that they are pre-installed in the ROM 52 or the like.

[0075] The program executed by the information processing device of the embodiment may be configured to be provided as a computer program product by being recorded in an installable or executable format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0076] Furthermore, the program executed by the information processing apparatus of the embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the information processing apparatus of the embodiment may be configured to be provided or distributed via a network such as the Internet.

[0077] The programs executed by the information processing device of the embodiment can cause the computer to function as each unit of the information processing device described above. In this computer, the CPU 51 can read the programs from a computer-readable storage medium onto a main storage device and execute the programs.

[0078] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0079] 100, 100-2 Information processing device 101 Reception 102, 102-2 Object model learning section 103 Student Model Learning Department 104 Output control section 110 Changes 111 Pruning Section 112 Generation part 120 Selection section 121 Teacher Model Selection Section 122 Initial parameter selection section 131 Storage section

Claims

1. a target model training unit for training a target model to be reduced in size; A modification unit that modifies the target model into a student model having a size smaller than that of the target model; a selection unit that selects, as a teacher model, one of a plurality of models including the target model and one or more intermediate models smaller in size than the target model, according to a comparison result between a size of the target model and a size of the student model; a student model learning unit that learns the student model by distillation using the selected teacher model; An information processing device comprising:

2. The selection unit further selects initial values ​​of the parameters of the student model from the parameters of the target model; The student model learning unit learns the student model using the selected initial value. The information processing device according to claim 1 .

3. the target model is a neural network model including a plurality of layers, each layer including a plurality of elements; the student model is a neural network model including a plurality of layers, each layer including a plurality of elements; the modification unit modifies the target model into the student model by deleting a part of an element included in the target model; The selection unit selects the initial value of a parameter of an element included in the student model from parameters of elements included in the target model or the intermediate model corresponding to the element included in the student model. The information processing device according to claim 2 .

4. the intermediate model is a neural network model including a plurality of layers, each layer including a plurality of elements; the selection unit selects, for each of the first layers representing the multiple layers included in the intermediate model, multiple element groups each including one or more elements included in the first layer, and selects, as the initial value, a parameter of an element included in the target model or the intermediate model corresponding to an element included in the element group having an evaluation value larger than that of the other element groups among the multiple element groups; The information processing device according to claim 3 .

5. The evaluation value is A value representing the accuracy of inference using the student model in which parameters of elements included in the target model or the intermediate model corresponding to the elements included in the element group are set as the initial values. The information processing device according to claim 4.

6. The evaluation value is When inference is performed on a plurality of data using the student model in which parameters of elements included in the target model or the intermediate model corresponding to elements included in the element group are set as the initial values ​​for each of the plurality of first layers, the smaller the difference between the output statistics of the corresponding layer included in the student model and the output statistics of the first layer, the larger the value. The information processing device according to claim 4.

7. The evaluation value is When a backward process is performed on a plurality of data using the student model in which parameters of elements included in the target model or the intermediate model corresponding to elements included in the element group are set as the initial values ​​for each of the plurality of first layers, the smaller the difference between the gradient statistics calculated for the corresponding layer included in the student model and the gradient statistics calculated for the first layer, the larger the value becomes. The information processing device according to claim 4.

8. the target model, the intermediate model and the student model are neural network models including a plurality of layers; The change unit is a pruning unit that generates the intermediate model by pruning the target model; A generation unit that generates the student model by increasing or decreasing the size of each of the layers included in the intermediate model by a specified ratio. The information processing device according to claim 1 .

9. The target model learning unit pre-learns the target model using a first data set for pre-learning, and learns the pre-learned target model using a second data set different from the first data set. The information processing device according to claim 1 .

10. a target model training unit for training a target model to be reduced in size; A modification unit that modifies the target model into a student model having a size smaller than that of the target model; A selection unit that selects an initial value of the parameter of the student model from the parameters of the target model; a student model learning unit that learns the student model using the selected initial value; An information processing device comprising:

11. An information processing method executed by an information processing device, a target model training step of training a target model to be reduced in size; A modification step of modifying the target model to a student model having a size smaller than that of the target model; a selection step of selecting, as a teacher model, one of a plurality of models including the target model and one or more intermediate models smaller in size than the target model, according to a comparison result between the size of the target model and the size of the student model; a student model learning step of learning the student model by distillation using the selected teacher model; An information processing method comprising:

12. On the computer, a target model training step of training a target model to be reduced in size; A modification step of modifying the target model to a student model having a size smaller than that of the target model; a selection step of selecting, as a teacher model, one of a plurality of models including the target model and one or more intermediate models smaller in size than the target model, according to a comparison result between the size of the target model and the size of the student model; a student model learning step of learning the student model by distillation using the selected teacher model; A program for executing the above.

Images