Neural network generation method

Abstract

A neural network generation method according to the present invention comprises: a parsing step in which a trained teacher neural network (TL) having an N-number of sub-networks is generated parsing a trained teacher neural network (TL) of M layers into N-number of sub-networks; and a training step in which a trained student neural network (SL) is generated by inputting a dataset into the trained teacher neural network (TL) and a student neural network (S) of N layers, respectively, and training the student neural network (S). In the training step, an N-number of teacher-side outputs, which are outputs for each of the N-number of sub-networks, and an N-number of student-side outputs, which are outputs of each of the N layers of the student neural network, are associated in order of processing from an input layer toward an output layer, and weighted data for each of the N layers in the associated order are determined to generate a trained student neural network (SL).

Description

Technical Field

[0001] The present disclosure relates to a neural network generation method for generating a learned student neural network.

Background Art

[0002] Conventionally, a method of learning a student neural network based on a learned teacher neural network to generate a learned student neural network is known.

[0003] Non-Patent Document 1 discloses a method for efficiently searching for a network architecture by structuring a teacher neural network in block units and searching for the loss between the teacher neural network and a plurality of candidate student neural networks. In this method, knowledge distillation is utilized to imitate the student neural network.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Generally, the teacher neural network is often a more complex model than the student neural network. In that case, it is possible to increase the complexity of the student neural network to increase the imitation degree of the student neural network, but due to the constraints of the student neural network, it may be difficult to increase the complexity.

[0006] Therefore, this disclosure provides a neural network generation method that can simply generate a trained student neural network. [Means for solving the problem]

[0008] To achieve the above objective, a neural network generation method according to one form of this disclosure is: A method for generating neural networks performed by a computer, The method includes a preparation step of preparing a trained teacher neural network consisting of M layers (where M is an integer of 3 or more) and a student neural network consisting of N layers (where N is an integer of 2 or more), fewer than M layers; a decomposition step of decomposing the trained teacher neural network into N subnetworks; and a learning step of inputting a dataset into each of the decomposed trained teacher neural network and the student neural network, and training the student neural network to generate a trained student neural network, wherein the decomposition step has multiple grouping patterns that change the decomposition position when decomposing the trained teacher neural network, and the learning step includes (1) the N (1) The N teacher outputs, which are the outputs of each subnetwork, and the N student outputs, which are the outputs of each of the N layers of the student neural network, are associated in processing order from the input layer to the output layer. (2) From among multiple combinations of the trained teacher neural network and the student neural network having multiple grouping patterns, the combination of the trained teacher neural network and the student neural network that produces the smallest evaluation value based on the respective errors between the associated N teacher outputs and N student outputs is selected. (3) Based on the selected student neural network, the weight data for each of the N layers of the student neural network is determined to generate the trained student neural network.

[0009] To achieve the above objective, a neural network generation method according to one form of this disclosure is: A method for generating neural networks performed by a computer,Preparation step: Prepare a trained teacher neural network consisting of M layers (where M is an integer greater than or equal to 3) and a student neural network consisting of fewer than M layers, N layers (where N is an integer greater than or equal to 2); Decomposition step: Decompose the trained teacher neural network so that it includes at least a first subnetwork and a second subnetwork in order from the input side; Input a dataset into the first subnetwork and the student neural network, respectively, and reduce the first error based on the error between the first teacher output, which is the output of the first subnetwork, and the first student output, which is the output of the first layer of the student neural network. A first decision step in which the weight data of the first layer is determined by training a student neural network; a second decision step in which the weight data of the second layer is determined by inputting a dataset into each of the student neural networks, which include a partial neural network composed of the first subnetwork and the second subnetwork, and a first layer and a second layer located after the first layer having the weight data determined in the first decision step, and training the student neural network so that the second error based on the error between the second teacher output, which is the output of the second subnetwork, and the second student output, which is the output of the second layer, is reduced; The decomposition step includes changing the decomposition position when decomposing the trained teacher neural network so that the first subnetwork and the second subnetwork each have multiple grouping patterns, the first decision step selects a combination of the first subnetwork and the first layer of the student neural network from among multiple combinations of the first subnetwork having multiple grouping patterns and the first layer of the student neural network that minimizes the first error, and determines the weight data of the first layer based on the selected first layer of the student neural network, the second decision step selects a combination of a plurality of sub-neural networks composed of the first subnetwork having multiple grouping patterns and the first and second layers of the student neural network that minimizes the second error, and determines the weight data of the second layer based on the selected second layer of the student neural network. [Effects of the Invention]

[0010] The neural network generation method of this disclosure makes it possible to simply generate a trained student neural network. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 shows an example of a teacher neural network and a student neural network. [Figure 2] Figure 2 schematically shows a trained teacher neural network and a student neural network before training. [Figure 3]FIG. 3 is a diagram showing the relationship between a learned teacher neural network and a student neural network. [Figure 4A] FIG. 4A is a diagram schematically showing the neural network generation method according to Embodiment 1. [Figure 4B] FIG. 4B is a diagram schematically showing the neural network generation method following FIG. 4A. [Figure 4C] FIG. 4C is a diagram schematically showing the neural network generation method following FIG. 4B. [Figure 5] FIG. 5 is a flowchart showing the neural network generation method according to Embodiment 1. [Figure 6] FIG. 6 is a diagram showing an example in which a coefficient is multiplied by the error between the teacher-side output and the student-side output. [Figure 7] FIG. 7 is a flowchart showing a method for deriving a coefficient to be multiplied by the error. [Figure 8] FIG. 8 is a diagram showing an example of a method for deriving a coefficient to be multiplied by the error. [Figure 9] FIG. 9 is a diagram showing an example when resizing a feature map. [Figure 10] FIG. 10 is a diagram showing another example when resizing a feature map. [Figure 11] FIG. 11 is a diagram schematically showing the neural network generation method according to Embodiment 2. [Figure 12] FIG. 12 is a flowchart showing the neural network generation method according to Embodiment 2. [Figure 13] FIG. 13 is a diagram showing an evaluation value based on the error between the teacher-side output and the student-side output. [Figure 14] FIG. 14 is a flowchart showing a method for deriving a coefficient to be multiplied by the error. [Figure 15] FIG. 15 is a diagram showing an example of a method for deriving a coefficient to be multiplied by the error.

MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that all the embodiments described below are specific examples of the present disclosure. The numerical values, shapes, materials, standards, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, the components not described in the independent claims indicating the highest-level concept of the present disclosure are described as optional components. Also, each figure is not necessarily drawn precisely. In each figure, substantially the same configuration is denoted by the same reference numeral, and redundant descriptions may be omitted or simplified.

[0013] [Basic Configuration of Neural Network] The basic configurations of the teacher neural network and the student neural network will be described.

[0014] FIG. 1 is a diagram showing an example of a teacher neural network and a student neural network.

[0015] Each neural network shown in FIG. 1 has a multi-layer structure and is composed of an input layer, a plurality of intermediate layers, and an output layer. Each of the input layer, the intermediate layer, and the output layer is, for example, a convolutional layer or a fully-connected layer, and has a plurality of nodes (not shown) corresponding to neurons.

[0016] Since the teacher neural network is composed of a complex inference model, the load when using the teacher neural network may become heavy. Therefore, a student neural network that mimics the teacher neural network is used.

[0017] The student neural network is a simplified inference model with fewer layers than the teacher neural network. The student neural network disclosed herein is a model for achieving processing comparable to that of the teacher neural network using fixed hardware such as a system LSI (Large Scale Integrated Circuit). The number of layers in the student neural network is predetermined according to the hardware configuration of the system LSI. On the other hand, the weight data for each layer of the system LSI corresponding to each layer of the student neural network is variable, and it is possible to implement the weight data into the system LSI later.

[0018] The neural network generation method disclosed herein generates a trained student neural network by training the student neural network under the constraint that the number of layers of the student neural network is predetermined, and then determining the weight data for each layer. For example, by implementing the weight data of the trained student neural network into a system LSI, processing comparable to that of a teacher neural network can be achieved with the above-mentioned system LSI.

[0019] To facilitate understanding of this disclosure, the teacher neural network and the student neural network are explained schematically as follows.

[0020] Figure 2 schematically shows the trained teacher neural network TL and the untrained student neural network S. Figure 2 shows a schematic representation of the neural networks in Figure 1.

[0021] The trained supervising neural network TL shown in Figure 2(a) consists of M layers (where M is an integer greater than or equal to 3). M is the number of layers when the trained supervising neural network TL is represented as a layer structure. In this example, the trained supervising neural network TL has 9 layers. For example, the first layer of the 9 layers is the input layer, and layers 2 through 9 are hidden layers. Note that all 9 layers may also be hidden layers.

[0022] The student neural network S shown in Figure 2(b) before training consists of N layers (where N is an integer greater than or equal to 2), which is fewer than M layers. N is the number of layers when the student neural network S is represented as a layer structure, and is predetermined by the hardware configuration, such as the system LSI. In this example, the student neural network S has three layers. For example, the first layer of the three layers is the input layer, and the second and third layers are hidden layers. Note that all three layers may also be hidden layers.

[0023] The following describes an embodiment in which a student neural network S, consisting of 3 layers, is trained based on a pre-trained teacher neural network TL consisting of 9 layers, and the weight data for each of the 3 layers is determined.

[0024] (Embodiment 1) [Outline of neural network generation method] Figure 3 shows the relationship between the trained teacher neural network TL and the student neural network S.

[0025] Figure 3 shows the three subnetworks included in the trained teacher neural network TL, and the three layers included in the untrained student neural network S. A subnetwork is a network that constitutes a part of a neural network. The reason for having three subnetworks is to match the number of layers in the student neural network S. Note that the grouping of the three subnetworks shown in the figure is merely an example.

[0026] Here, the three subnetworks are referred to as the first subnetwork T1, the second subnetwork T2, and the third subnetwork T3, in the order of processing from the input layer to the output layer. Similarly, the three layers included in the student neural network S are referred to as the first layer S1, the second layer S2, and the third layer S3, in the order of processing from the input layer to the output layer. In this example, the first subnetwork T1, the second subnetwork T2, and the third subnetwork T3 are associated with the first layer S1, the second layer S2, and the third layer S3, in the order they appear from the input layer to the output layer.

[0027] For example, the number of grouping options when generating three subnetworks from a nine-layer neural network is the same as the number of options when selecting two decomposition positions from eight decomposition positions p1 to p8 located between each of the nine layers. Therefore, the total number of grouping patterns when generating three subnetworks is 28 (8C2 = 28).

[0028] In Embodiment 1, instead of exploring all 28 grouping patterns, only a portion of the 28 patterns are explored to determine the weight data for the three layers of the student neural network S.

[0029] Figures 4A, 4B, and 4C are schematic diagrams illustrating the neural network generation method according to Embodiment 1. Figure 4A shows the search for the first subnetwork T1, Figure 4B shows the search for the second subnetwork T2, and Figure 4C shows the search for the third subnetwork T3.

[0030] First, as shown in Figure 4A, we perform a search on the first subnetwork T1. Since the second subnetwork T2 and the third subnetwork T3, which follow the first subnetwork T1, each require at least one layer, the first subnetwork T1 is formed with 7 layers or less (9 layers minus 2 layers). In other words, as shown in Figure 4A, the first subnetwork T1 can take on 7 patterns when decomposed at decomposition positions p1, p2, p3, p4, p5, p6, or p7.

[0031] Next, a training dataset containing input data and labels is input to both the trained teacher neural network TL and the student neural network S. The number of inputs in the dataset can be 100 or 1000. The student neural network S is then trained to minimize the first error e1, which is based on the error between the first teacher output to1 (the output of the first subnetwork T1 and subsequent layers) and the first student output so1 (the output of the first layer S1 and subsequent layers of the student neural network S). Here, the first error e1 is defined as (the error between the output of the first subnetwork T1 alone and the output of the first layer S1 of the student neural network S alone). The above training is performed for each of the seven patterns, and the pattern that minimizes the first error e1 is selected from among the seven patterns.

[0032] In this example, the first error e1 is smallest when the decomposition is performed at decomposition position p3, and as shown in Figure 4A, the first subnetwork T1 is determined to match the pattern obtained when the decomposition is performed at decomposition position p3. Furthermore, the weight data for the first layer S1 of the student neural network S is determined to match the weight data W1 obtained from training the first subnetwork T1 and the first layer S1 when the decomposition is performed at decomposition position p3.

[0033] Next, as shown in Figure 4B, we perform a search for the second subnetwork T2. The search for the second subnetwork T2 is performed on the assumption that the first subnetwork T1 is fixed at the previously determined decomposition position p3. Since the third subnetwork T3, which follows the second subnetwork T2, requires at least one layer, the second subnetwork T2 is formed of five layers or less, which is the six layers other than the first subnetwork T1 minus one layer. In other words, as shown in Figure 4B, the second subnetwork T2 can take on five patterns when decomposed at decomposition positions p4, p5, p6, p7, or p8.

[0034] Next, the dataset is input to both the trained teacher neural network TL, which includes the first subnetwork T1 and the second subnetwork T2, and the student neural network S, which includes the first layer S1 with the previously determined weight data W1 and the second layer S2 located after the first layer S1. Then, the student neural network S is trained to minimize the second error e2, which is based on the error between the second teacher output to2 (the output of the second subnetwork T2 and beyond) and the second student output so2 (the output of the second layer S2 and beyond of the student neural network S). Here, the second error e2 = first error e1 + (error between the output of the second subnetwork T2 alone and the output of the second layer S2 of the student neural network S alone). The above training is performed for each of the five patterns, and the pattern that minimizes the second error e2 is selected from the five patterns.

[0035] In this example, the second error e2 is smallest when decomposed at decomposition position p6, and as shown in Figure 4B, the second subnetwork T2 is determined to be the pattern obtained when decomposed at decomposition position p6. In addition, the weight data of the second layer S2 of the student neural network S is determined to be the weight data W2 obtained from training the first subnetwork T1 when decomposed at decomposition position p3, the second subnetwork T2 when decomposed at decomposition position p6, and the first layer S1 and the second layer S2.

[0036] Next, as shown in Figure 4C, we perform a search for the third subnetwork T3. The search for the third subnetwork T3 is performed on the premise that the first subnetwork T1 is fixed at the previously determined decomposition position p3 and the second subnetwork T2 is fixed at the previously determined decomposition position p6. The third subnetwork T3 is formed by the three layers other than the first subnetwork T1 and the second subnetwork T2. In other words, the third subnetwork T3 can take one pattern when decomposed at decomposition position p6, as shown in Figure 4C.

[0037] Next, the dataset is input to each of the following: the trained teacher neural network TL, which includes the first subnetwork T1, the second subnetwork T2, and the third subnetwork T3; and the student neural network S, which includes the first layer S1 with weight data W1, the second layer S2 with weight data W2, and the third layer S3 located after the second layer S2. Then, the student neural network S is trained to minimize the third error e3, which is based on the error between the third teacher output to3 (output of the third subnetwork T3 and beyond) and the third student output so3 (output of the third layer S3 and beyond of the student neural network S). Here, the third error e3 = second error e2 + (error between the output of the third subnetwork T3 alone and the output of the third layer S3 of the student neural network S alone).

[0038] In the example shown in Figure 4C, the weight data for the third layer S3 of the student neural network S is determined by the weight data W3 obtained from training the pre-trained teacher neural network TL with the first layer S1, the second layer S2, and the third layer S3. This determines the weight data W1, W2, and W3 corresponding to the first layer S1, the second layer S2, and the third layer S3, respectively, and generates the pre-trained student neural network SL.

[0039] Thus, in Embodiment 1, the three teacher-side outputs, which are the outputs of each of the three subnetworks, and the three student-side outputs, which are the outputs of each of the three layers of the student neural network S, are mapped in processing order from the input layer to the output layer. Then, by determining the weight data W1 to W3 for each of the three layers of the student neural network S in the mapped order, a trained student neural network SL is generated. This method allows for the simple generation of a trained student neural network SL with minimal processing load. For example, in the above example, the total number of searches is 7 + 5 + 1 = 13, which is a reduction in the number of searches compared to performing a full search of all 28 patterns.

[0040] [Flowchart of Neural Network Generation Method] The flow of how to generate a neural network will be explained with reference to Figure 5.

[0041] Figure 5 is a flowchart showing the neural network generation method according to Embodiment 1.

[0042] The neural network generation method according to Embodiment 1 includes a preparation step S100, a decomposition step S200, and a learning step S300. The learning step S300 includes a first decision step S310, a second decision step S320, and a third decision step S330.

[0043] Preparation step S100 is the step of preparing a trained teacher neural network TL consisting of M layers, and a student neural network S consisting of N layers, which is fewer than M layers.

[0044] Decomposition step S200 is a step in which the trained teacher neural network TL is decomposed so that it includes at least a first subnetwork T1 and a second subnetwork T2, starting from the input side. Specifically, in decomposition step S200, the first subnetwork T1 and the second subnetwork T2, each having multiple grouping patterns, are generated by changing the decomposition position when decomposing the trained teacher neural network TL. Furthermore, decomposition step S200 generates a third subnetwork T3 located after the second subnetwork T2 by changing the decomposition position when decomposing the trained teacher neural network TL.

[0045] The decomposition step S200 is performed as needed before the first decision step S310, the second decision step S320, and the third decision step S330. For example, in this example, the first subnetwork T1 is decomposed and extracted before the first decision step S310, the second subnetwork T2 is decomposed and extracted before the second decision step S320, and the third subnetwork T3 is decomposed and extracted before the third decision step S330.

[0046] The first decision step S310 is the step of determining the weight data W1 of the first layer S1 of the student neural network S. In the first decision step S310, the training dataset containing input data and labels is input to both the trained teacher neural network TL and the student neural network S. Then, the weight data of the first layer S1 is determined by training the student neural network S so that the first error e1, which is based on the error (or loss value) between the first teacher output to1, which is the output of the first subnetwork T1, and the first student output so1, which is the output of the first layer S1 of the student neural network S, is minimized.

[0047] Specifically, in the first decision step S310, from among multiple combinations of the first subnetwork T1 having multiple grouping patterns and the first layer S1 of the student neural network S, the combination of the first subnetwork T1 and the first layer S1 of the student neural network S that minimizes the first error e1 is selected. Then, based on the first layer S1 of the student neural network S after the selection, the weight data W1 of the first layer S1 is determined.

[0048] The second decision step S320 is the step of determining the weight data W2 for the second layer S2 of the student neural network S. In the second decision step S320, the dataset is input to the trained teacher neural network TL, which includes the first subnetwork T1 and the second subnetwork T2, and to the student neural network S, which includes the first layer S1 and the second layer S2 located after the first layer S1, each having the weight data W1 determined in the first decision step S310. Then, the weight data W2 for the second layer S2 is determined by training the student neural network S so that the second error e2, which is based on the error (or loss value) between the second teacher output to2, which is the output of the second subnetwork T2, and the second student output so2, which is the output of the second layer S2 of the student neural network S, becomes small.

[0049] Specifically, in the second decision step S320, from among the combinations of multiple partial neural networks consisting of a first subnetwork T1 that minimizes the first error e1 and a second subnetwork T2 having multiple grouping patterns, and the first layer S1 and second layer S2 of the student neural network S, the combination of the partial neural network that minimizes the second error e2 and the first layer S1 and second layer S2 of the student neural network S is selected. Then, based on the second layer S2 of the student neural network S after the selection, the weight data W2 of the second layer S2 is determined.

[0050] The third decision step S330 is the step of determining the weight data W3 for the third layer S3 of the student neural network S. In the third decision step S330, the dataset is input to the trained teacher neural network TL, which includes the first subnetwork T1, the second subnetwork T2, and the third subnetwork T3, as well as to the student neural network S, which includes the first layer S1 with the weight data W1 determined in the first decision step S310, the second layer S2 with the weight data W2 determined in the second decision step S320, and the third layer S3 located after the second layer S2. Then, the weight data W3 for the third layer S3 is determined by training the student neural network S so that the third error e3, which is based on the error (or loss value) between the third teacher output to3, which is the output of the third subnetwork T3, and the third student output so3, which is the output of the third layer S3 of the student neural network S, becomes small.

[0051] By performing these steps S100 to S300, the trained student neural network SL can be generated simply with minimal processing load.

[0052] Furthermore, if in the decomposition step S200 the trained teacher neural network TL can be decomposed to generate a third subnetwork T3 having multiple grouping patterns, that is, if other subnetworks different from the third subnetwork T3 can be generated, the third decision step S330 may be performed as shown below.

[0053] In this case, the third decision step S330 selects a combination of a multiple partial neural network, consisting of a first subnetwork T1 that minimizes the first error e1, a second subnetwork T2 that minimizes the second error e2, and a third subnetwork T3 having multiple grouping patterns, and the first layer S1, second layer S2, and third layer S3 of the student neural network, that minimizes the third error e3. Then, based on the selected third layer S3 of the student neural network, the weight data W3 of the third layer S3 is determined.

[0054] Furthermore, while it is desirable that the dataset used in the learning step S300 described above be the same dataset, it does not necessarily have to contain the same input data and labels. The dataset may be a superdataset containing all input data and labels, or a subdataset containing some representative input data and labels. For example, the teacher neural network may be trained by inputting the teacher training data, which is the dataset used for teacher training, and the student neural network S may be trained using a portion of the teacher training data. In other words, the dataset used in the learning step S300 may consist of a subdataset that is part of the teacher training data. In this case, the student neural network S may also be trained using the teacher training data.

[0055] [Modification 1 of Embodiment 1] A modified example of Embodiment 1 will be described with reference to Figures 6 to 8.

[0056] Embodiment 1 described an example in which the student neural network S is trained to minimize the error between the teacher's output and the student's output. However, it is not limited to this, and it is also possible to multiply the error by a coefficient and train the student neural network S to minimize the error after multiplying by the coefficient. Therefore, Modification 1 describes a method for deriving the coefficient to be multiplied by the error.

[0057] Figure 6 shows an example where a coefficient is applied to the error between the teacher's output and the student's output.

[0058] Figure 6(a) shows the first error e1, which is the value obtained by multiplying the error between the output of the first subnetwork T1 and the output of the first layer S1 by the coefficient k1. Figure 6(b) shows the second error e2, which is the value obtained by multiplying the error between the output of the second subnetwork T2 and the output of the second layer S2 by the coefficient k2 and adding it to the first error e1. Figure 6(c) shows the third error e3, which is the value obtained by multiplying the error between the output of the third subnetwork T3 and the output of the third layer S3 by the coefficient k3 and adding it to the second error e2.

[0059] Each error may also be a loss value, which is the difference between the teacher's output and the student's output. Each coefficient k1, k2, and k3 is a value that indicates the importance of the error in each output; the larger the coefficient value, the more important the error in that output.

[0060] In this example, each coefficient is derived based on the sensitivity of the behavior of the target neural network. Note that each coefficient is pre-determined according to the error during the preparation stage before executing the neural network generation method flow shown in Figure 5.

[0061] Figure 7 is a flowchart showing the method for deriving the coefficient multiplied by the error. Figure 8 is a diagram showing an example of the method for deriving the coefficient multiplied by the error.

[0062] As shown in Figure 7, the method for deriving the coefficients includes the steps of preparing a reference training neural network Tr, generating a reference training neural network Tr having N subnetworks, and deriving coefficients to be multiplied by the error.

[0063] In the step of preparing the reference training neural network Tr, a reference training neural network Tr is prepared that has noisy weight data, which is obtained by adding noise to the weight data corresponding to each layer of the trained training neural network TL.

[0064] In the step of generating a reference training neural network Tr having N subnetworks, the reference training neural network Tr is decomposed into N subnetworks to generate a reference training neural network Tr having N subnetworks.

[0065] In the step of deriving the coefficients, the dataset is input into both the trained training neural network TL and the reference training neural network Tr. The loss values ​​between the outputs of the corresponding layers of the trained training neural network TL and the reference training neural network Tr are used to calculate the sum of the noise-induced fluctuations in the loss values ​​for each of the N subnetworks, and the coefficients are set based on the relative magnitudes of these sums.

[0066] Specifically, in the step of deriving the coefficients, as shown in Figure 8(a), the loss fluctuation value ΔL for each layer of the trained teacher neural network TL is measured by comparing it with the weights (Z) of each layer of the trained teacher neural network TL plus noise (n) (Z+n). Then, the coefficients are set to larger values ​​starting from the subnetworks corresponding to the larger loss fluctuation value ΔL. For example, as shown in Figure 8(b), the loss fluctuation values ​​ΔL for each layer in each subnetwork T1, T2, and T3 are summed to derive the coefficients k1, k2, and k3. By multiplying the error by the coefficients obtained in this way, it becomes possible to evaluate the error (or loss) between the teacher output and the student output according to the sensitivity of the neural network's behavior.

[0067] [Modification 2 of Embodiment 1] A modified example of Embodiment 1, Part 2, will be described with reference to Figures 9 and 10.

[0068] Embodiment 1 showed an example of calculating the error by simply comparing the teacher's output and the student's output. However, it is not limited to this, and it is also possible to calculate the error after making the size of the feature maps of each subnetwork the same as the feature maps of each layer of the student neural network. Therefore, Modification 2 describes an example in which the feature maps are resized to the same size.

[0069] Figure 9 shows an example of resizing a feature map.

[0070] Figure 9(a) shows an example where the feature maps of the subnetwork are larger than the feature maps of each layer of the student neural network. In this example, as shown in Figure 9(b), the feature maps of the subnetwork are reduced in size to match the feature maps of each layer of the student neural network. Methods for resizing include the same method as pooling calculations in convolutional neural networks (CNNs) or methods such as the interpolation kernel (Bi-Linear) used in image resizing. By resizing in this way, it becomes possible to accurately calculate the error between the teacher's output and the student's output.

[0071] Figure 10 shows another example of resizing a feature map.

[0072] Figure 10(a) shows an example where the feature maps of the subnetwork are larger than the feature maps of each layer of the student neural network. In this example, as shown in Figure 10(b), the feature maps of the student neural network are enlarged to the same size as the feature maps of the subnetwork. Methods for resizing include, for example, the same method as upsampling calculations in convolutional neural networks (CNNs) or interpolation kernels (Bi-Linear) used in image resizing. By resizing in this way, it becomes possible to accurately determine the error between the teacher's output and the student's output.

[0073] For example, in the first decision step S310 shown in Figure 5, the size of one feature map of the first teacher-side output to1 and the first student-side output so1 may be converted to match the size of the other feature map, and then a loss calculation may be performed to obtain the first error e1. In the second decision step S320, the size of one feature map of the second teacher-side output to2 and the second student-side output so2 may be converted to match the size of the other feature map, and then a loss calculation may be performed to obtain the second error e2. In the third decision step S330, the size of one feature map of the third teacher-side output to3 and the third student-side output so3 may be converted to match the size of the other feature map, and then a loss calculation may be performed to obtain the third error e3.

[0074] (Embodiment 2) [Outline of neural network generation method] Embodiment 2 describes an example of performing a brute-force search on the first subnetwork T1, the second subnetwork T2, and the third subnetwork T3. In Embodiment 2, as well as in other embodiments, the teacher neural network and the student neural network will be explained schematically to facilitate understanding of this disclosure.

[0075] Figure 11 is a schematic diagram illustrating the neural network generation method according to Embodiment 2.

[0076] For example, when decomposing a trained supervising neural network TL consisting of M layers to generate N subnetworks, the grouping pattern is ( M-1 C N-1 It is represented as ( ). In this example, M=9 and N=3, so the grouping pattern is ( 9-1 C 3-1 This results in the pattern. In other words, in Embodiment 2, all 28 grouping patterns are explored to determine the weight data for the three layers of the student neural network S.

[0077] In Embodiment 2, a full search is performed on the first subnetwork T1, the second subnetwork T2, and the third subnetwork T3. The first subnetwork T1, the second subnetwork T2, and the third subnetwork T3 can take on 28 patterns when decomposed at decomposition positions p1, p2, p3, p4, p5, p6, p7, and p8. Note that the notation "Decomposition positions p1, p2 → p1, p8" shown in Figure 11 indicates that decomposition position p1 was fixed, and the other decomposition position was changed from decomposition position p2 to p8, resulting in a total of 7 patterns being searched. The same applies to the notations for the other decomposition positions.

[0078] Next, a training dataset containing input data and labels is input to both the trained teacher neural network TL and the student neural network S. The number of inputs in the dataset can be 100 or 1000. The student neural network S is then trained to minimize the evaluation value v, which is based on the error between the teacher output to (the output of the trained teacher neural network TL) and the student output so (the output of the student neural network S). This training is performed for each of the 28 patterns, and the pattern that produces the smallest evaluation value v is selected from among the 28 patterns.

[0079] In this example, the evaluation value v is smallest when decomposed at decomposition positions p3 and p6. As shown in Figure 11, the first subnetwork T1 is determined to match the pattern when decomposed at decomposition position p3, and the second subnetwork T2 and third subnetwork T3 are determined to match the patterns when decomposed at decomposition position p6. Furthermore, the weight data for each layer of the student neural network S is the weight data when decomposed at decomposition positions p3 and p6. The weight data for the first layer S1 is determined to be W1, the weight data for the second layer S2 is determined to be W2, and the weight data for the third layer S3 is determined to be W3.

[0080] In Embodiment 2, a fully searched set of three subnetworks is performed to determine the weight data W1 to W3 for each of the three layers of the student neural network S, thereby generating a trained student neural network SL. This method allows for the accurate and simple generation of a trained student neural network SL.

[0081] [Flowchart of Neural Network Generation Method] The flow of how to generate a neural network will be explained with reference to Figure 12.

[0082] Figure 12 is a flowchart showing the neural network generation method according to Embodiment 2.

[0083] The neural network generation method according to Embodiment 2 includes a preparation step S100, a decomposition step S200, and a learning step S300.

[0084] Preparation step S100 is the step of preparing a trained teacher neural network TL consisting of M layers, and a student neural network S consisting of N layers, which is fewer than M layers.

[0085] The decomposition step S200 is a step in which a trained teacher neural network TL is decomposed into N subnetworks to generate a trained teacher neural network TL having N subnetworks. In the decomposition step S200, a trained teacher neural network TL having multiple grouping patterns is generated by changing the decomposition position when decomposing the trained teacher neural network TL.

[0086] The learning step S300 is a step in which a dataset is input into both the trained teacher neural network TL, which has N subnetworks, and the student neural network S, and the student neural network S is trained to generate a trained student neural network SL.

[0087] Specifically, in learning step S300, first, the N teacher outputs, which are the outputs of each of the N subnetworks, and the N student outputs, which are the outputs of each of the N layers of the student neural network S, are mapped in processing order from the input layer to the output layer. Next, from among multiple combinations of the trained teacher neural network TL, which has multiple grouping patterns, and the student neural network S, the combination of the trained teacher neural network TL and the student neural network S that produces the smallest evaluation value v based on the error between the mapped teacher outputs and student outputs is selected. Then, based on the selected student neural network S, the weight data for each of the N layers of the student neural network S is determined, thereby generating the trained student neural network SL.

[0088] By performing these steps S100 to S300, a pre-trained student neural network SL can be generated accurately and simply.

[0089] [Modification 1 of Embodiment 2] A modified example of Embodiment 2, Part 1, will be described with reference to Figures 13 to 15.

[0090] In Embodiment 2, an example was described in which the student neural network S is trained so that the evaluation value v becomes small. However, it is not limited to this, and the student neural network S can also be trained so that the evaluation value v after multiplying the error by a coefficient becomes small. Therefore, in this Modification 1, a method for deriving the evaluation value v will be described.

[0091] Figure 13 shows the evaluation values ​​based on the error between the teacher's output and the student's output.

[0092] Figure 13 shows the evaluation value v, which is obtained by multiplying the error between the individual output of the first subnetwork T1 and the individual output of the first layer S1 by a coefficient k1, multiplying the error between the individual output of the second subnetwork T2 and the individual output of the second layer S2 by a coefficient k2, and multiplying the error between the individual output of the third subnetwork T3 and the individual output of the third layer S3 by a coefficient k3, and then summing these results. In other words, the evaluation value v is the value obtained by multiplying each of the N errors (the errors between the N teacher-side outputs and the N student-side outputs) by the coefficient corresponding to each of the N errors, and then summing them up.

[0093] Each error may also be a loss value, which is the difference between the teacher's output and the student's output. Each coefficient k1, k2, and k3 is a value that indicates the importance of the error in each output; the larger the coefficient value, the more important the error in that output.

[0094] In this example as well, each coefficient is derived based on the sensitivity of the behavior of the target neural network. Each coefficient is pre-determined according to the error during the preparation stage before executing the neural network generation method flow shown in Figure 12.

[0095] Figure 14 is a flowchart showing the method for deriving the coefficient multiplied by the error. Figure 15 is a diagram showing an example of the method for deriving the coefficient multiplied by the error.

[0096] As shown in Figure 14, the method for deriving the coefficients includes the steps of preparing a reference training neural network Tr, generating a reference training neural network Tr having N subnetworks, and deriving coefficients to be multiplied by the error.

[0097] In the step of preparing the reference training neural network Tr, a reference training neural network Tr is prepared that has noisy weight data, which is obtained by adding noise to the weight data corresponding to each layer of the trained training neural network TL.

[0098] In the step of generating a reference training neural network Tr having N subnetworks, the reference training neural network Tr is decomposed into N subnetworks to generate a reference training neural network Tr having N subnetworks.

[0099] In the step of deriving the coefficients, the dataset is input into both the trained training neural network TL and the reference training neural network Tr. The loss values ​​between the outputs of the corresponding layers of the trained training neural network TL and the reference training neural network Tr are used to calculate the sum of the noise-induced fluctuations in the loss values ​​for each of the N subnetworks, and the coefficients are set based on the relative magnitudes of these sums.

[0100] Specifically, in the step of deriving the coefficients, as shown in Figure 15(a), the loss fluctuation value ΔL for each layer of the trained teacher neural network TL is measured by comparing it with the weights (Z) of each layer of the trained teacher neural network TL plus noise (n) (Z+n). Then, the coefficients are set to larger values ​​starting from the subnetworks corresponding to the larger loss fluctuation value ΔL. For example, as shown in Figure 15(b), the loss fluctuation values ​​ΔL for each layer in each subnetwork are summed to derive the coefficients k1, k2, and k3. By multiplying the error by the coefficients obtained in this way, it becomes possible to obtain an evaluation value v based on the error between the teacher output and the student output, depending on the sensitivity of the neural network's behavior.

[0101] In the above example, we showed how to obtain the evaluation value v by simply comparing the teacher's output and the student's output. However, we are not limited to this, and we can also obtain the evaluation value v after making the size of the feature maps of each subnetwork the same as the feature maps of each layer of the student neural network. The method for resizing the feature maps to make them the same size is the same as in Modification 2 of Embodiment 1.

[0102] (summary) A neural network generation method according to one embodiment of the present disclosure includes a preparation step of preparing a trained teacher neural network TL consisting of M layers and a student neural network S consisting of N layers, fewer than M layers; a decomposition step of decomposing the trained teacher neural network TL into N subnetworks; and a learning step of inputting a dataset into each of the decomposed trained teacher neural network TL and student neural network S, and training the student neural network S to generate a trained student neural network SL. The learning step generates a trained student neural network SL by associating the N teacher-side outputs, which are the outputs of each of the N subnetworks, with the N student-side outputs, which are the outputs of each of the N layers of the student neural network S, in processing order from the input layer to the output layer, and determining the weight data for each of the N layers of the student neural network S in the order of association.

[0103] In this way, by determining the weight data for each of the N layers of the student neural network S in the processing order from the input layer to the output layer, a trained student neural network SL can be generated simply with low processing load. Furthermore, for example, generally, the weights of each layer are learned using random values ​​as initial values, but since the behavior of the output tends to change in response to the input, it is thought that determining the input weights first, as in this disclosure, increases the efficiency of generating a trained student neural network.

[0104] Alternatively, the learning step may involve training the student neural network S so that the error between each of the N teacher outputs and the N student outputs is minimized, thereby determining the weight data.

[0105] By training the student neural network S in a way that minimizes the above-mentioned errors, the weight data for each layer can be accurately determined.

[0106] A neural network generation method according to one embodiment of the present disclosure includes a preparation step of preparing a trained teacher neural network TL composed of M layers and a student neural network S composed of N layers, which is fewer than M layers; a decomposition step of decomposing the trained teacher neural network TL into N subnetworks; and a learning step of inputting a dataset into each of the decomposed trained teacher neural network TL and student neural network S, and training the student neural network S to generate a trained student neural network SL. The decomposition step has multiple grouping patterns that change the decomposition position when decomposing the trained teacher neural network TL. The learning steps are as follows: (1) The N teacher outputs, which are the outputs of each of the N subnetworks, and the N student outputs, which are the outputs of each of the N layers of the student neural network S, are mapped in the order of processing from the input layer to the output layer; (2) From among multiple combinations of the trained teacher neural network TL and the student neural network S, which have multiple grouping patterns, the combination of the trained teacher neural network TL and the student neural network S that produces the smallest evaluation value v based on the respective errors between the mapped N teacher outputs and N student outputs is selected; and (3) Based on the selected student neural network S, the weight data for each of the N layers of the student neural network S is determined, thereby generating a trained student neural network SL.

[0107] In this way, by selecting the combination that minimizes the evaluation value v from among multiple combinations, the weight data for each layer of the student neural network S can be determined with high accuracy. This allows for the accurate and simple generation of a pre-trained student neural network SL.

[0108] Alternatively, the evaluation value v may be the sum of the N errors, which are the respective errors between the N teacher-side outputs and the N student-side outputs, multiplied by a coefficient corresponding to each of the N errors.

[0109] In this way, by multiplying each of the N errors by a coefficient corresponding to that error, an evaluation value v can be generated that corresponds to the importance of the error. Therefore, a trained student neural network SL can be generated that has weight data corresponding to the evaluation value.

[0110] Furthermore, the neural network generation method further includes the steps of: preparing a reference teacher neural network Tr having noisy weight data obtained by adding noise to the weight data corresponding to each layer of the trained teacher neural network TL; decomposing the reference teacher neural network Tr into N subnetworks; and deriving coefficients corresponding to each of the N errors based on the trained teacher neural network TL and the reference teacher neural network Tr. The step of deriving the coefficients may involve inputting the dataset into both the trained teacher neural network TL and the reference teacher neural network Tr, using the loss values ​​between the outputs of the corresponding layers of the trained teacher neural network TL and the reference teacher neural network Tr to find the sum of the noise-induced fluctuations in the loss values ​​for each of the N subnetworks, and deriving the coefficients based on the relative magnitudes of the sums.

[0111] According to this, it becomes possible to determine an evaluation value v between the teacher's output and the student's output, depending on the sensitivity of the neural network's behavior. This allows for the accurate determination of the weight data for each layer of the student neural network S.

[0112] Alternatively, during the learning step, the size of one feature map of the teacher's output and the student's output may be converted to match the size of the other feature map before performing a loss calculation to determine the error.

[0113] By matching the size of the feature maps in this way, it becomes possible to accurately determine the error between the teacher's output and the student's output. This allows for accurate determination of the weight data for each layer of the student neural network S.

[0114] Furthermore, the training neural network is trained using training data, and the dataset may consist of a portion of the training data.

[0115] According to this method, a pre-trained student neural network (SL) can be generated quickly and simply.

[0116] Furthermore, the neural network generation method may also include the step of training a student neural network S using teacher training data.

[0117] This can improve the reliability of the trained student neural network SL.

[0118] A neural network generation method according to one embodiment of this disclosure includes a preparation step of preparing a trained teacher neural network TL consisting of M layers and a student neural network S consisting of N layers, fewer than M layers; a decomposition step of decomposing the trained teacher neural network TL so that it includes at least a first subnetwork T1 and a second subnetwork T2 in order from the input side; and inputting a dataset into the first subnetwork T1 and the student neural network S, respectively, and reducing the first error e1 based on the error between the first teacher output to1, which is the output of the first subnetwork T1, and the first student output so1, which is the output of the first layer S1 of the student neural network S. The method includes a first decision step of determining the weight data W1 of the first layer S1 by training a student neural network S, and a second decision step of inputting a dataset into each of the student neural networks S, which includes a first neural network composed of a first subnetwork T1 and a second subnetwork T2, and the first layer S1 and the second layer S2 located after the first layer S1, which have the weight data W1 determined in the first decision step, and training the student neural network S to minimize the second error e2, which is based on the error between the second teacher output to2, which is the output of the second subnetwork T2, and the second student output so2, which is the output of the second layer S2, thereby determining the weight data W2 of the second layer S2.

[0119] In this way, by determining the weight data for each of the N layers of the student neural network S sequentially from the input side, the trained student neural network SL can be generated simply with minimal processing load.

[0120] Furthermore, in the decomposition step, by changing the decomposition position when decomposing the trained teacher neural network TL, the first subnetwork T1 and the second subnetwork T2 each have multiple grouping patterns. The first decision step selects a combination of the first subnetwork T1 and the first layer S1 of the student neural network S from among multiple combinations of the first subnetwork T1 having multiple grouping patterns and the first layer S1 of the student neural network S that minimizes the first error e1, and then determines the weight data W1 of the first layer S1 of the student neural network S based on the selected first layer S1 of the student neural network S. The second decision step may involve selecting a combination of a sub-neural network and the first and second layers S1 and S2 of the student neural network S that minimizes the second error e2, from among multiple sub-neural networks consisting of a first subnetwork T1 that minimizes the first error e1 and a second subnetwork T2 having multiple grouping patterns, and the first and second layers S1 and S2 of the student neural network S, and then determining the weight data W2 of the second layer S2 of the second layer S2 of the student neural network S based on the selected second layer S2.

[0121] In this way, by selecting the combination that minimizes the error from among multiple combinations, the weight data for each layer of the student neural network S can be determined with high accuracy.

[0122] Alternatively, in the first decision step, the size of one feature map of the first teacher output to1 and the first student output so1 may be converted to match the size of the other feature map, and then a loss calculation may be performed to obtain the first error e1. In the second decision step, the size of one feature map of the second teacher output to2 and the second student output so2 may be converted to match the size of the other feature map, and then a loss calculation may be performed to obtain the second error e2.

[0123] By matching the size of the feature maps in this way, it becomes possible to accurately determine the error between the teacher's output and the student's output. This allows for accurate determination of the weight data for each layer of the student neural network S.

[0124] The decomposition step also includes a third subnetwork T3 located after the second subnetwork T2. The neural network generation method further includes a third decision step performed after the second decision step. The third decision step may determine the weight data W3 for the third layer S3 by inputting the dataset into each of the student neural network S, which includes a partial neural network consisting of a first subnetwork T1, a second subnetwork T2, and a third subnetwork T3, as well as a first layer S1 having weight data W1 determined in the first decision step, a second layer S2 having weight data W2 determined in the second decision step, and a third layer S3 located after the second layer S2. The student neural network S is then trained to minimize a third error e3 based on the error between a third teacher output to3, which is the output of the third subnetwork T3, and a third student output so3, which is the output of the third layer S3 of the student neural network S.

[0125] In this way, by determining the weight data for each of the N layers of the student neural network S sequentially from the input side, the trained student neural network SL can be generated simply with minimal processing load.

[0126] Furthermore, in the decomposition step, by changing the decomposition position when decomposing the trained teacher neural network TL, the first subnetwork T1 and the second subnetwork T2 each have multiple grouping patterns. The first decision step selects a combination of the first subnetwork T1 and the first layer S1 of the student neural network S from among multiple combinations of the first subnetwork T1 having multiple grouping patterns and the first layer S1 of the student neural network S that minimizes the first error e1, and then determines the weight data W1 of the first layer S1 of the student neural network S based on the selected first layer S1 of the student neural network S. The second decision step selects a combination of a partial neural network consisting of a first subnetwork T1 and a second subnetwork T2 having multiple grouping patterns, which minimizes the first error e1, and the first layer S1 and second layer S2 of the student neural network S, which minimizes the second error e2. Based on the selected second layer S2 of the student neural network S, the weight data W2 of the second layer S2 is determined. Furthermore, in the decomposition step, by changing the decomposition position when decomposing the trained teacher neural network TL, if the third subnetwork T3 has multiple grouping patterns, the third decision step may select a combination of the sub-neural network and the first layer S1, second layer S2, and third layer S3 of the student neural network S that minimizes the third error e3, from among the combinations of the sub-neural network composed of multiple sub-neural networks (T1 when the first error e1 is minimized, T2 when the second error e2 is minimized, and T3 when the third subnetwork has multiple grouping patterns) and the first layer S1, second layer S2, and third layer S3 of the student neural network S, and then determine the weight data W3 of the third layer S3 based on the selected third layer S3 of the student neural network S.

[0127] In this way, by selecting the combination that minimizes the error from among multiple combinations, the weight data for each layer of the student neural network S can be determined with high accuracy.

[0128] Alternatively, in the first decision step, the size of one feature map of the first teacher output to1 and the first student output so1 may be converted to match the size of the other feature map, and then a loss calculation may be performed to obtain the first error e1; in the second decision step, the size of one feature map of the second teacher output to2 and the second student output so2 may be converted to match the size of the other feature map, and then a loss calculation may be performed to obtain the second error e2; and in the third decision step, the size of one feature map of the third teacher output to3 and the third student output so3 may be converted to match the size of the other feature map, and then a loss calculation may be performed to obtain the third error e3.

[0129] By matching the size of the feature maps in this way, it becomes possible to accurately determine the error between the teacher's output and the student's output. This allows for accurate determination of the weight data for each layer of the student neural network S.

[0130] Furthermore, the training neural network is trained using training data, and the dataset may consist of a portion of the training data.

[0131] According to this method, a pre-trained student neural network (SL) can be generated quickly and simply.

[0132] Furthermore, the neural network generation method may also include the step of training a student neural network S using teacher training data.

[0133] This can improve the reliability of the trained student neural network SL.

[0134] (Other embodiments) The neural network generation method relating to this disclosure has been described above based on each embodiment, but this disclosure is not limited to these embodiments. As long as it does not deviate from the spirit of this disclosure, various modifications that a person skilled in the art can conceive of will be applied to each embodiment, and other forms constructed by combining some of the components of each embodiment are also included within the scope of this disclosure.

[0135] In the first modification of Embodiment 1 and the second modification of Embodiment 2, examples were shown where the coefficients are determined according to the sensitivity of the neural network's behavior, but this method is not the only way. For example, since the input side of a convolutional neural network mainly involves feature extraction that does not depend on the input, the behavior becomes more sensitive as it approaches the output side. Therefore, the coefficients may be set to increase as it approaches the output side of the neural network (for example, k1 ≤ k2 ≤ k3).

[0136] Furthermore, the following forms may also be included within the scope of one or more aspects of this disclosure.

[0137] (1) The disclosure may be the methods described above. Alternatively, it may be a computer program that implements these methods using a computer, or a digital signal consisting of the computer program.

[0138] (2) The Disclosure also relates to a computer system comprising a microprocessor and memory, wherein the memory stores the computer program, and the microprocessor operates in accordance with the computer program.

[0139] (3) Alternatively, the program or the digital signal may be carried out by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network or the like.

[0140] (4) The above embodiments and the above modifications may be combined. [Industrial applicability]

[0141] This disclosure can be widely used for neural network generation methods that mimic trained teacher neural networks to generate trained student neural networks. [Explanation of symbols]

[0142] e1, e2, e3 error p1, p2, p3, p4, p5, p6, p7, p8 Decomposition position S Student Neural Network S1 1st layer S2 2nd layer S3 3rd layer SL (Student Learning) Neural Network so, so1, so2, so3 Student-side output T1 Subnetwork 1 T2 Second Subnetwork T3 Third Subnetwork TL (Trained Supervisor Neural Network) Tr Reference Training Neural Network to, to1, to2, to3 teacher side output v Evaluation value W1, W2, W3 weight data

Claims

1. A method for generating a neural network performed by a computer, Preparation steps include preparing a trained teacher neural network consisting of M layers (where M is an integer greater than or equal to 3), and a student neural network consisting of fewer than M layers, N layers (where N is an integer greater than or equal to 2), The decomposition step involves decomposing the aforementioned trained teacher neural network into N subnetworks, A learning step to generate a trained student neural network by inputting a dataset into each of the trained teacher neural network and the student neural network, which are decomposed into the N subnetworks, and training the student neural network; Includes, The decomposition step has multiple grouping patterns that change the decomposition position when decomposing the trained teacher neural network, The aforementioned learning steps are: (1) The N teacher-side outputs, which are the outputs of each of the N subnetworks, and the N student-side outputs, which are the outputs of each of the N layers of the student neural network, are associated in processing order from the input layer to the output layer. (2) From among multiple combinations of the trained teacher neural network having multiple grouping patterns and the student neural network, select the combination of the trained teacher neural network and the student neural network that produces the smallest evaluation value based on the respective errors between the N corresponding teacher-side outputs and the N corresponding student-side outputs. (3) The trained student neural network is generated by determining the weight data for each of the N layers of the student neural network based on the selected student neural network. Methods for generating neural networks.

2. The evaluation value is obtained by multiplying the N errors, which are the respective errors between the N teacher-side outputs and the N student-side outputs, by a coefficient corresponding to each of the N errors, and then summing them up. The neural network generation method according to claim 1.

3. moreover, The steps include preparing a reference training neural network having noisy weight data, which is obtained by adding noise to the weight data corresponding to each layer of the aforementioned trained training training neural network, The steps include: decomposing the aforementioned reference training neural network into N subnetworks; The steps include: deriving the coefficients corresponding to each of the N errors based on the trained training neural network and the reference training neural network; Includes, The step of deriving the coefficient is: The dataset is input to both the pre-trained training neural network and the reference training neural network, and the sum of the noise-induced fluctuations in the loss values ​​for each of the N subnetworks is calculated using the loss values ​​between the outputs of the corresponding layers of the pre-trained training neural network and the reference training neural network. The coefficients are then derived based on the relative magnitudes of these sums. The neural network generation method according to claim 2.

4. In the learning step described above, the size of one of the feature maps of the teacher's output and the student's output is converted to match the size of the other feature map, and then the loss calculation is performed to determine the error. A method for generating a neural network according to any one of claims 1 to 3.

5. The aforementioned training neural network is trained using training data. The aforementioned dataset is composed of a portion of the aforementioned training data. A method for generating a neural network according to any one of claims 1 to 3.

6. moreover, This includes the step of training the student neural network using the aforementioned teacher training data. The neural network generation method according to claim 5.

7. A method for generating a neural network performed by a computer, Preparation steps include preparing a trained teacher neural network consisting of M layers (where M is an integer greater than or equal to 3), and a student neural network consisting of fewer than M layers, N layers (where N is an integer greater than or equal to 2), The decomposition step involves decomposing the aforementioned trained teacher neural network so that it includes at least a first subnetwork and a second subnetwork, in order from the input side. A first decision step involves inputting a dataset into the first subnetwork and the student neural network, and training the student neural network to minimize the first error, which is based on the error between the first teacher-side output, which is the output of the first subnetwork, and the first student-side output, which is the output of the first layer of the student neural network; thereby determining the weight data for the first layer. A second decision step involves inputting a dataset into each of the sub-neural networks composed of the first subnetwork and the second subnetwork, and the student neural network, which includes a first layer having the weight data determined in the first decision step and a second layer located after the first layer, and training the student neural network to reduce the second error based on the error between the second teacher output, which is the output of the second subnetwork, and the second student output, which is the output of the second layer, thereby determining the weight data of the second layer. Includes, In the decomposition step, by changing the decomposition position when decomposing the trained teacher neural network, the first subnetwork and the second subnetwork each have multiple grouping patterns. The first decision step selects a combination of the first subnetwork and the first layer of the student neural network from among a plurality of combinations of the first subnetwork having a plurality of grouping patterns and the first layer of the student neural network that minimizes the first error, and determines the weight data of the first layer based on the first layer of the student neural network after selection. The second decision step selects a combination of a plurality of partial neural networks, each comprising the first subnetwork and the second subnetwork having multiple grouping patterns, that minimizes the first error, from among the combinations of the first subnetwork and the first and second layers of the student neural network, that minimizes the second error, and then determines the weight data of the second layer based on the second layer of the student neural network after selection. Methods for generating neural networks.

8. In the first decision step, the size of one of the feature maps of the first teacher output and the first student output is converted to match the size of the other feature map, and then a loss calculation is performed to obtain the first error. In the second decision step, the size of one of the feature maps of the second teacher output and the second student output is converted to match the size of the other feature map, and then a loss calculation is performed to determine the second error. The neural network generation method according to claim 7.

9. The decomposition step includes a third subnetwork located downstream of the second subnetwork, Furthermore, it includes a third decision step that is performed after the second decision step, The 3rd decision step is, The weight data for the third layer is determined by inputting the dataset into each of the following: a partial neural network composed of the first subnetwork, the second subnetwork, and the third subnetwork, and the student neural network which includes the first layer having the weight data determined in the first decision step, the second layer having the weight data determined in the second decision step, and the third layer located after the second layer; and training the student neural network to minimize the third error, which is based on the error between the third teacher output, which is the output of the third subnetwork, and the third student output, which is the output of the third layer of the student neural network. The neural network generation method according to claim 7.

10. In the decomposition step, by changing the decomposition position when decomposing the trained teacher neural network, the first subnetwork and the second subnetwork each have multiple grouping patterns. The first decision step selects a combination of the first subnetwork and the first layer of the student neural network from among a plurality of combinations of the first subnetwork having a plurality of grouping patterns and the first layer of the student neural network that minimizes the first error, and determines the weight data of the first layer based on the first layer of the student neural network after selection. The second decision step selects a combination of a plurality of partial neural networks, each comprising the first subnetwork and the second subnetwork having multiple grouping patterns, that minimizes the first error, and the first and second layers of the student neural network, and determines the weight data of the second layer based on the second layer of the student neural network after selection. Furthermore, in the decomposition step, by changing the decomposition position when decomposing the trained teacher neural network, if the third subnetwork has multiple grouping patterns, The 3rd decision step is, From among the combinations of a plurality of partial neural networks, each comprising the first subnetwork that minimizes the first error, the second subnetwork that minimizes the second error, and the third subnetwork having multiple grouping patterns, the combination of the partial neural network and the first, second, and third layers of the student neural network that minimizes the third error is selected, and the weight data of the third layer is determined based on the selected third layer of the student neural network. The neural network generation method according to claim 9.

11. In the first decision step, the size of one of the feature maps of the first teacher output and the first student output is converted to match the size of the other feature map, and then a loss calculation is performed to obtain the first error. In the second decision step, the size of one of the feature maps of the second teacher output and the second student output is converted to match the size of the other feature map, and then a loss calculation is performed to obtain the second error. In the third decision step, the size of one of the feature maps of the third teacher output and the third student output is converted to match the size of the other feature map, and then a loss calculation is performed to determine the third error. A method for generating a neural network according to claim 9 or 10.

12. The aforementioned training neural network is trained using training data. The aforementioned dataset is composed of a portion of the aforementioned training data. A method for generating a neural network according to any one of claims 7, 9, or 10.

13. moreover, This includes the step of training the student neural network using the aforementioned teacher training data. The neural network generation method according to claim 12.

Images