Information processing device, information processing method, and program

Abstract

To further accurately execute learning by distillation of a model reduced in size.SOLUTION: An information processing device includes a target model learning unit, a change unit, a selection unit, and a student model learning unit. The target model learning unit learns a target model to be reduced in size. The change unit changes the target model to a student model having a smaller size than the target model. The selection unit selects, as a teacher model, any one of a plurality of models including the target model and one or more intermediate models smaller in size than the target model in accordance with comparison results between the size of the target model and the size of the student model. The student model learning unit learns the student model by distillation using the selected teacher model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In neural networks such as deep neural networks (DNNs), in consideration of quickly training a model in an environment where computing resources such as edge terminals are limited, processing for reducing the size of the model may be performed in order to reduce the amount of computation and storage capacity of the model. Further, in order to improve the accuracy of the model after size reduction, a technique for training a student model, which is the model after size reduction, by distillation (also referred to as knowledge distillation) using the model before size reduction as a teacher model is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

[0005] The present invention aims to provide an information processing device, an information processing method, and a program that can perform learning by distillation of a smaller model with higher accuracy. [Means for solving the problem]

[0006] The information processing device of this embodiment comprises a target model learning unit, a modification unit, a selection unit, and a student model learning unit. The target model learning unit learns the target model to be reduced in size. The modification unit changes the target model to a student model that is smaller in size than the target model. The selection unit selects one of a plurality of models, including the target model and one or more intermediate models that are smaller in size than the target model, as the teacher model, based on the result of comparing the size of the target model and the size of the student model. The student model learning unit learns the student model by distillation using the selected teacher model. [Brief explanation of the drawing]

[0007] [Figure 1] Block diagram of an information processing device according to an embodiment. [Figure 2] A diagram illustrating an example of changing the target model to a student model. [Figure 3] A diagram illustrating an example of generating an intermediate model and a student model from a target model. [Figure 4] Block diagram of the modified section. [Figure 5] A diagram illustrating an example of a student model. [Figure 6] Block diagram of the selected section. [Figure 7] A diagram illustrating the overview of the initial parameter selection process. [Figure 8] A diagram illustrating an example of a process for finding combinations. [Figure 9] A schematic diagram illustrating the selection of combinations using evaluation values. [Figure 10] A schematic diagram illustrating the selection of combinations using evaluation values. [Figure 11] A schematic diagram illustrating the selection of combinations using evaluation values. [Figure 12] Flowchart of the learning process in the embodiment. [Figure 13] A block diagram of a modified information processing device. [Figure 14] Hardware configuration diagram of the information processing device according to the embodiment. [Modes for carrying out the invention]

[0008] A preferred embodiment of the information processing device according to this invention will be described in detail below with reference to the attached drawings.

[0009] In the process of reducing the size of a model, the ratio of the student model's size to the teacher model's size (the size reduction rate) can vary. For example, if the reduction rate is large, the accuracy of learning through distillation may decrease due to the large difference between the student model and the teacher model.

[0010] The following embodiments select a teacher model according to the size of a student model (model after size reduction) learned by distillation. Also, the initial values of parameters (initial parameters) during learning of the student model are selected from the parameters of a model to be reduced in size (hereinafter referred to as the target model). Thereby, learning by distillation of a model with a reduced size can be performed with higher accuracy.

[0011] Also, in the prior art, when pre-training is required for the target model, pre-training is also required for the student model after size reduction. In the embodiments, by appropriately selecting the initial parameters, pre-training of the student model can be made unnecessary. Thereby, the learning cost can be reduced.

[0012] In the embodiments, according to the ratio of the sizes of the student model and the target model, any one of a plurality of models including the target model and one or more intermediate models is selected as the teacher model. The intermediate model is a model smaller in size than the target model. Also, the initial parameters of the student model are selected from the initial parameters of the target model based on an evaluation value.

[0013] Note that hereinafter, the case where the model (target model, intermediate model, student model) is a neural network model (hereinafter also simply referred to as a neural network) will be described as an example. The model is not limited to this, and may be a model having any other structure. A neural network includes a plurality of layers each including a plurality of elements.

[0014] For example, when the neural network is a convolutional neural network (CNN), the plurality of elements may be a plurality of channels included in the convolutional layer. The elements are not limited to this, and may be any other elements. For example, the elements may be nodes included in each layer of the neural network. Hereinafter, the case where the channels are the elements will be described as an example.

[0015] Figure 1 is a block diagram showing an example of the configuration of an information processing device 100 according to an embodiment. As shown in Figure 1, the information processing device 100 includes a reception unit 101, a target model learning unit 102, a student model learning unit 103, an output control unit 104, a modification unit 110, a selection unit 120, and a storage unit 131.

[0016] The reception unit 101 receives input of various types of information used by the information processing device 100. For example, the reception unit 101 receives information indicating the target model (e.g., parameters such as weights and biases), and multiple training data (training datasets) used to train the model.

[0017] The target model learning unit 102 learns the target model that is to be reduced in size. For example, the target model learning unit 102 learns the target model using a training dataset.

[0018] The modification unit 110 changes the target model to a student model, which is smaller in size than the target model. The student model is a model that is learned by distillation using the teacher model. For example, the modification unit 110 changes the target model to a student model by deleting some of the channels contained in each of the multiple layers contained in the target model. Note that changing the target model to a student model can also be interpreted as equivalent to generating a student model from the target model.

[0019] The method used to modify the student model by the modification unit 110 can be any method, but for example, the methods shown in (M1) to (M3) below can be applied. Note that each of the methods shown below can generate one or more intermediate models along with the student model.

[0020] (M1) A student model is generated by uniformly reducing the number of channels in each of the multiple layers included in the target model. For example, the number of channels in each layer of the target model is reduced by a specified ratio (hereinafter also called scale). Hereafter, changing (increasing or decreasing) the number of channels by a specified scale may be referred to as scaling.

[0021] The channels to be reduced by scaling can be selected in any way. For example, random selection or sequential selection of a number of channels corresponding to the scale from a specific position (such as the beginning, middle, or end) can be applied.

[0022] Figure 2 shows an example of changing the target model 201 to the student model 202 using (M1). The target model 201 contains four layers L1 to L4. The student model 202 contains four layers L1-2 to L4-2. The numbers listed below each layer indicate the number of channels contained in the corresponding layer. Figure 2 shows an example where a scale of 0.5 is specified. As shown in Figure 2, the number of channels in each of the four layers contained in the target model 201 is reduced according to the scale of 0.5, and the student model 202 is generated.

[0023] The modification unit 110 may change the target model into multiple models according to multiple scales, designating one of the modified models as the student model and the rest as intermediate models. For example, the modification unit 110 may scale the target model into an intermediate model using a scale of 0.75, and then scale the target model into a student model using a scale of 0.5. Thus, the intermediate model may be larger in size than the student model. The intermediate model may be the same size as the student model, or smaller in size than the student model.

[0024] (M2) A student model is generated by pruning the target model. Pruning is a process that, for example, identifies operations that can be reduced among the multiple operations implemented by the target model and generates a model with those operations reduced. Any pruning method can be used, but for example, the methods described in Non-Patent Document 1 and Non-Patent Document 2 can be applied.

[0025] The modification unit 110 may generate models of multiple sizes by changing the pruning intensity, designate one of the generated models as the student model, and use the rest as intermediate models. For example, in a method that calculates importance for each channel, as in Non-Patent Document 1, the intensity can be manipulated by changing the threshold (importance threshold) used to determine which channels to reduce. In a method that uses compactification to learn a sparse model with many zeros in its weights, as in Non-Patent Document 2, the intensity can be manipulated by changing the learning parameters (e.g., weight decay).

[0026] Figure 3 shows an example of generating an intermediate model 301 and a student model 302 from the target model 201 using (M2). The intermediate model 301 includes four layers L1-3 to L4-3. The student model 302 includes four layers L1-4 to L4-4. The intermediate model 301 corresponds to a model generated with low intensity. The student model 302 corresponds to a model generated with higher intensity than the intermediate model 301. Because the intensity is relatively low, the degree of accuracy degradation of the intermediate model 301 is relatively smaller than that of the student model 302.

[0027] (M3) Pruning the target model generates one or more intermediate models, and scaling one of the generated intermediate models generates the student model.

[0028] Figure 4 is a block diagram showing an example configuration of the modification unit 110 when (M3) is applied. As shown in Figure 2, the modification unit 110 includes a pruning unit 111 and a generation unit 112.

[0029] The pruning unit 111 generates one or more intermediate models by pruning the target model. The generation unit 112 generates a student model by scaling the size of each of the multiple layers contained in one of the intermediate models by a specified scale.

[0030] Figure 5 shows an example of student model 501 generated by (M3). Student model 501 includes four layers L1-5 to L4-5. The numbers at the top of each layer indicate the number of channels before and after pruning in the corresponding layer. The number to the right of the symbol " / " indicates the number of channels before pruning, and the number to the left of the symbol " / " indicates the number of channels after pruning. The numbers at the bottom of each layer indicate the number of channels after scaling in the corresponding layer. In this example, the scale is 0.5.

[0031] Returning to Figure 1, the selection unit 120 selects the teacher model to be used for learning the student model by distillation. The selection unit 120 also selects the initial values ​​(initial parameters) of the student model's learning parameters. These parameters are, for example, weights and biases.

[0032] Figure 6 is a block diagram showing a detailed example of the functional configuration of the selection unit 120. As shown in Figure 6, the selection unit 120 includes a teacher model selection unit 121 and an initial parameter selection unit 122.

[0033] The teacher model selection unit 121 selects one of several models, including the target model and one or more intermediate models, as the teacher model, for example, based on the result of comparing the size of the target model with the size of the student models. For example, the teacher model selection unit 121 calculates the ratio of the size of the student models to the size of the target model (= size of the student model / size of the target model). The size can be calculated in any way, but for example, it can be calculated using the size of the model parameters (data size) and the number of operations (FLOPs) during inference using the model.

[0034] The teacher model selection unit 121 compares the calculated ratio with a threshold. If the ratio is greater than the threshold, it selects the target model as the teacher model. If the ratio is less than or equal to the threshold, it selects an intermediate model as the teacher model. If two or more intermediate models are generated, the teacher model selection unit 121 may use multiple thresholds to select an intermediate model of a size corresponding to the magnitude of the ratio. The thresholds are predetermined, for example, according to the type of target model.

[0035] The initial parameter selection unit 122 selects the initial parameters (initial values) of multiple layers included in the student model from the parameters of multiple layers included in the target model.

[0036] Initial parameters are, for example, initial values ​​for parameters set for one or more channels to be included in each layer at the start of learning. For example, the initial parameter selection unit 122 selects initial parameters for one or more channels to be included in each of the multiple layers included in the student model from the parameters of multiple channels included in the multiple layers included in the target model.

[0037] The parameters of multiple channels contained in multiple layers included in the target model may be the parameters of the target model before training (initial parameter values) or the parameters after training the target model.

[0038] For example, the following method can be applied to select the initial parameters. First, the initial parameter selection unit 122 determines multiple combinations (multiple element groups) for each of the multiple layers (first layer) included in the intermediate model, including some channels selected from the multiple channels included in that layer. The initial parameter selection unit 122 selects the channels included in the combinations from which the evaluation value is greater than that of the other combinations. For the selected channels, the initial parameter selection unit 122 selects the parameters set for the corresponding channels in the corresponding layers of the target model as the initial parameters.

[0039] Figure 7 is a diagram illustrating the overview of the initial parameter selection process. Figure 7 shows an example of selecting initial parameters for layer L3, one of the four layers L1 to L4 included in the student model. Similar processing is performed for the other layers L1, L2, and L4.

[0040] Furthermore, Figure 7 shows an example where the target model 701 has 8 channels in layer L3, and an intermediate model 702 with 6 channels in layer L3 is generated through pruning. The initial parameter selection unit 122 finds multiple combinations containing 3 channels from the 6 channels included in layer L3 of the intermediate model 702.

[0041] Figure 8 shows an example of finding three combinations. The numbers in the intermediate model 702 are information that identifies the channels. The initial parameter selection unit 122 selects three mutually distinct channels from the six channels of the intermediate model 702 to find three combinations 703-1, 703-2, and 703-3. Note that the number of combinations is not limited to three; there may be two, four or more.

[0042] Returning to Figure 7, the initial parameter selection unit 122 selects three channels from the combination with the largest evaluation value among the three combinations. The model containing the three selected channels corresponds to the student model 703. The initial parameter selection unit 122 sets the initial parameters for each channel included in layer L3 of the student model 703 by selecting them from the parameters of the corresponding channels in layer L3 of the target model 701. The parameters selected at this time may be the parameters of the target model 701 before training (initial parameter values), or the parameters after the target model 701 has been trained. The initial parameter selection unit 122 may also set the initial parameters by selecting them from the parameters of the corresponding channels in layer L3 of the pruned intermediate model 702.

[0043] The evaluation value is used to assess whether the channels included in each combination are appropriate, and indicators such as those shown in (E1) to (E3) below can be used. In the following, a student model in which the parameters of the target model corresponding to the channels included in the combination are set as initial values ​​is called a candidate model. The evaluation value is calculated for each candidate model. (E1) A value representing the accuracy of inference using the candidate model (e.g., the rate of correct inference). (E2) A value that increases as the difference between the output statistics of the layers included in the candidate model and the output statistics of the corresponding layers included in the intermediate model becomes smaller when inference using the candidate model is performed on multiple data sets. (E3) A value that increases as the difference between the gradient statistics calculated for the layers included in the candidate model and the gradient statistics calculated for the corresponding layers included in the intermediate model, when backward processing using the candidate model is performed on multiple data sets, becomes smaller.

[0044] The statistics used in (E2) and (E3) can be any values, such as the mean, variance, maximum, and minimum. A value combining two or more statistics may be used as the evaluation value. For example, the sum of the absolute values ​​or sum of the squares of the differences between multiple statistics may be used as the evaluation value. The backward processing performed in (E3) is the process of calculating the gradient (derivative) of each layer with respect to the model's error, moving from the output layer of the model towards the input layer.

[0045] Figures 9 to 11 schematically illustrate the selection of combinations (student models) using evaluation values ​​(E1) to (E3), respectively.

[0046] Figure 9 shows examples of evaluation values ​​(accuracies) for each combination 703-1, 703-2, and 703-3. For example, the initial parameter selection unit 122 trains candidate models corresponding to each combination using the training data. The number of training iterations at this time should be sufficient to allow comparison of the accuracy of the trained candidate models. The initial parameter selection unit 122 performs inference for each trained candidate model and calculates the accuracy of the inference (such as the correctness rate) as an evaluation value. In the example in Figure 9, the initial parameter selection unit 122 selects combination 703-3, which corresponds to the candidate model with the highest evaluation value.

[0047] In Figure 10, the initial parameter selection unit 122 performs inference for each candidate model using multiple data points. For each layer, the initial parameter selection unit 122 compares the output statistics of the layer when inference is performed on multiple data points with the output statistics of the corresponding layer of the intermediate model. The initial parameter selection unit 122 selects the student model (candidate model) with the closest statistics (smallest difference).

[0048] In Figure 11, the initial parameter selection unit 122 performs inference and backward processing for each candidate model using multiple data points. For each layer, the initial parameter selection unit 122 compares the gradient statistics calculated by backward processing on multiple data points with the gradient statistics calculated for the corresponding layer of the intermediate model. The initial parameter selection unit 122 selects the student model (candidate model) with the closest statistics (smallest difference).

[0049] The initial parameter selection unit 122 selects the initial parameters for each layer of the student model corresponding to the selected combination, as shown in Figures 9 to 11, from the parameters of the corresponding layers included in the target model.

[0050] Returning to Figure 1, the student model learning unit 103 learns the student model by distillation using the teacher model selected by the teacher model selection unit 121. At this time, the student model learning unit 103 learns the student model using the initial parameters selected by the initial parameter selection unit 122.

[0051] The output control unit 104 controls the output of various types of information used by the information processing device 100. For example, the output control unit 104 outputs information (such as parameters) indicating a trained student model to an external device that performs inference using the student model.

[0052] Each of the above components (reception unit 101, target model learning unit 102, student model learning unit 103, output control unit 104, modification unit 110, and selection unit 120) can be implemented by, for example, one or more processors. For example, each of the above components may be implemented by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, i.e., by software. Each of the above components may also be implemented by a dedicated IC (Integrated Circuit) or other processor, i.e., by hardware. Each of the above components may also be implemented by using a combination of software and hardware. When multiple processors are used, each processor may implement one of the above components, or two or more of the above components.

[0053] The memory unit 131 stores various types of information used by the information processing device 100. For example, the memory unit 131 stores information received by the reception unit 101 (information indicating the target model, training data, etc.), and information indicating the trained student model.

[0054] The storage unit 131 can be composed of any commonly used storage medium, such as flash memory, memory card, RAM (Random Access Memory), HDD (Hard Disk Drive), and optical disc.

[0055] The information processing device 100 may be composed of one physical device or multiple physical devices. For example, the information processing device 100 may be built on a cloud environment. Furthermore, each part of the information processing device 100 may be distributed and provided on multiple devices.

[0056] Next, the learning process by the information processing device 100 of this embodiment will be described. The learning process involves training a student model, which has been reduced in size from the target model, using distillation with the teacher model. Figure 12 is a flowchart showing an example of the learning process in this embodiment.

[0057] The target model learning unit 102 trains the target model using the training dataset (step S101). The modification unit 110 changes the size of the trained target model and generates a student model (step S102).

[0058] The selection unit 120 (teacher model selection unit 121) selects either the target model or the intermediate model as the teacher model to be used for training the student model (step S103). For example, the teacher model selection unit 121 calculates the ratio of the student model size to the target model size, and selects the target model as the teacher model if the ratio is greater than a threshold. If the ratio is less than or equal to the threshold, the teacher model selection unit 121 selects the intermediate model as the teacher model.

[0059] The selection unit 120 (initial parameter selection unit 122) selects the initial values ​​(initial parameters) of the parameters to be used for training the student model (step S104).

[0060] The student model learning unit 103 learns the student model using the selected initial parameters and distillation using the selected teacher model (step S105).

[0061] The output control unit 104 outputs information (such as parameters) indicating the learned student model (step S106), and terminates the learning process.

[0062] In the explanation so far, the selection unit 120 has performed both the selection of a teacher model (teacher model selection unit 121) and the selection of initial parameters (initial parameter selection unit 122). The information processing device 100 may be configured to perform only one of these two tasks.

[0063] If the system has a feature that selects a teacher model considering the size of the student model, then, for example, if there is a large difference between the student model and the target model, an intermediate model with a smaller difference can be selected as the teacher model. This can suppress the decrease in learning accuracy caused by differences between the student model and the teacher model.

[0064] Furthermore, if the system has a function to select the initial parameters of the student model from the parameters of the target model, it becomes possible to set more appropriate initial parameters compared to methods such as initializing the parameters with random numbers. This makes it possible to train the student model with higher accuracy.

[0065] Thus, the information processing device of this embodiment can perform learning by distilling a smaller model with higher accuracy.

[0066] (modified version) The information processing device may further include a function for pre-training a target model. The modified example describes an example that includes a pre-training function. Figure 13 is a block diagram showing an example of the configuration of the modified information processing device 100-2. As shown in Figure 13, the information processing device 100-2 includes a reception unit 101, a target model learning unit 102-2, a student model learning unit 103, an output control unit 104, a modification unit 110, a selection unit 120, and a storage unit 131.

[0067] In the modified example, the function of the target model learning unit 102-2 differs from that of the above embodiment. The other configurations and functions are the same as those in Figure 1, which is a block diagram of the information processing device 100 of the above embodiment, so they are denoted by the same reference numerals and their explanation is omitted here.

[0068] The target model learning unit 102-2 differs from the target model learning unit 102 of the above embodiment in that it further performs pre-training of the target model. For example, the target model learning unit 102-2 pre-trains the target model using the pre-training training dataset DS_A (first dataset), and then trains the pre-trained target model using a different training dataset DS_B (second dataset).

[0069] The training dataset DS_A is a large, pre-prepared dataset that, for example, allows for the learning of general features and is easily transferable to other tasks. The training dataset DS_B is a dataset used to train a model that is better suited to a specific task.

[0070] If pre-training is performed, the initial parameter selection unit 122 may select the parameters of the target model after pre-training as the initial parameters of the student model. This makes it possible to eliminate the need for pre-training of the student model, even when using a target model that requires pre-training. This reduces the cost of training.

[0071] Furthermore, the information processing devices of the above embodiments and modified examples can be applied to learning models used in various systems. The system can be any system that includes a function using a model, but for example, it may be a system that uses a model on an edge terminal with a small memory capacity, as described below. A detection system in which a server device detects objects such as people (people flow) using information detected by an edge terminal (for example, information detected from images obtained by an imaging device (camera)). A control system in which a control device uses information detected by an edge terminal (for example, information detected from sensor data obtained from a sensor) to control equipment to be controlled (lighting equipment, elevators, etc.).

[0072] Next, the hardware configuration of the information processing device of the embodiment will be described using Figure 14. Figure 14 is an explanatory diagram showing an example of the hardware configuration of the information processing device of the embodiment.

[0073] The information processing device of this embodiment includes a control device such as a CPU 51, a storage device such as a ROM (Read Only Memory) 52 and RAM 53, a communication I / F 54 that connects to a network for communication, and a bus 61 that connects each part.

[0074] The program to be executed in the information processing device of this embodiment is provided pre-installed in a ROM 52 or the like.

[0075] The program executed by the information processing device of this embodiment may be configured to be provided as a computer program product by recording it in an installable or executable file format onto a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0076] Furthermore, the program executed by the information processing device of the embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the information processing device of the embodiment may be provided or distributed via a network such as the Internet.

[0077] The program executed in the information processing device of this embodiment can cause the computer to function as a component of the information processing device described above. This computer can read the program from a computer-readable storage medium onto the main memory and execute it using the CPU 51.

[0078] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0079] 100, 100-2 Information Processing Device 101 Reception Department 102, 102-2 Target Model Learning Unit 103 Student Model Learning Department 104 Output Control Unit 110 Changes 111 Pruning Department 112 Generation part 120 Selection Section 121 Teacher Model Selection Section 122 Initial parameter selection section 131 Storage section

Claims

1. A target model learning unit that trains the target model to be reduced in size, A modification unit that changes the aforementioned target model to a student model that is smaller in size than the aforementioned target model, A selection unit selects one of a plurality of models, including the target model and one or more intermediate models smaller in size than the target model, as the teacher model, based on the result of comparing the size of the target model with the size of the student model. A student model learning unit that learns the student model by distillation using the selected teacher model, An information processing device equipped with the following features.

2. The selection unit further selects initial values ​​for the student model parameters from the target model parameters, The student model learning unit learns the student model using the selected initial values. The information processing apparatus according to claim 1.

3. The aforementioned target model is a neural network model that includes multiple layers, each containing multiple elements. The aforementioned student model is a neural network model that includes multiple layers, each containing multiple elements. The modification part changes the target model to the student model by deleting some of the elements included in the target model. The selection unit selects the initial values ​​of the parameters of the elements included in the student model from the parameters of the elements included in the target model or the intermediate model that correspond to the elements included in the student model. The information processing apparatus according to claim 2.

4. The aforementioned intermediate model is a neural network model that includes multiple layers, each containing multiple elements. The selection unit selects a plurality of element groups for each of the first layers representing the plurality of layers included in the intermediate model, each of which includes one or more elements included in the first layer, and selects the parameters of the elements included in the target model or the intermediate model that correspond to the elements included in the element group whose evaluation value is greater than that of the other element groups, as the initial values. The information processing apparatus according to claim 3.

5. The aforementioned evaluation value is, This value represents the accuracy of inference using the student model, in which the parameters of the elements included in the target model or the intermediate model, corresponding to the elements included in the aforementioned group of elements, are set as the initial values. The information processing apparatus according to claim 4.

6. The aforementioned evaluation value is, For each of the multiple first layers, when inference is performed on multiple data points using the student model, in which the parameters of the elements included in the target model or the intermediate model corresponding to the elements included in the element group are set as the initial values, the value increases as the difference between the output statistics of the corresponding layer included in the student model and the output statistics of the first layer becomes smaller. The information processing apparatus according to claim 4.

7. The aforementioned evaluation value is, The value increases as the difference between the gradient statistics calculated for the corresponding layer in the student model and the gradient statistics calculated for the first layer becomes smaller when backward processing is performed on multiple data sets using the student model, in which the parameters of the elements included in the target model or the intermediate model corresponding to the elements included in the element group are set as the initial values ​​for each of the multiple first layers. The information processing apparatus according to claim 4.

8. The aforementioned target model, intermediate model, and student model are neural network models comprising multiple layers. The aforementioned modified part is, A pruning unit that generates the intermediate model by pruning the target model, The system includes a generation unit that generates the student model by increasing or decreasing the size of each of the multiple layers included in the intermediate model by a specified ratio. The information processing apparatus according to claim 1.

9. The target model learning unit pre-trains the target model using a first dataset for pre-training, and then trains the pre-trained target model using a second dataset different from the first dataset. The information processing apparatus according to claim 1.

10. An information processing method performed by an information processing device, The target model training step involves training the target model that will be subject to size reduction, A modification step of changing the aforementioned target model to a student model that is smaller in size than the aforementioned target model, A selection step in which, based on the result of comparing the size of the target model with the size of the student model, one of a plurality of models including the target model and one or more intermediate models smaller in size than the target model is selected as the teacher model. A student model learning step in which the student model is learned by distillation using the selected teacher model, Information processing methods including

11. On the computer, The target model training step involves training the target model that will be subject to size reduction, A modification step of changing the aforementioned target model to a student model that is smaller in size than the aforementioned target model, A selection step in which, based on the result of comparing the size of the target model with the size of the student model, one of a plurality of models including the target model and one or more intermediate models smaller in size than the target model is selected as the teacher model. A student model learning step in which the student model is learned by distillation using the selected teacher model, A program to execute.

Images