Learning prediction device, learning prediction method, and learning prediction program

Abstract

To support the training of a model.SOLUTION: A learning prediction device according to an embodiment includes a calculation unit, a determination unit, and an output unit. The calculation unit calculates multi-dimensional data characterizing a machine learning model and related data related to the multi-dimensional data. The determination unit generates support information for supporting the determination of the learning situation of the machine learning model by a dataset on the basis of whether the multi-dimensional data and the related data conform to prescribed criteria. The output unit outputs the support information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to a learning prediction device, a learning prediction method, and a learning prediction program. [Background technology]

[0002] In machine learning, there is a need to predict learning performance. As an example of predicting learning performance, it is sufficient to estimate the generalization error and determine the generalization performance. If the number of input data is sufficiently large compared to the number of parameters to be estimated, it is possible to estimate the generalization error (generalization performance) based on the central limit theorem in classical statistics, for example. However, in general machine learning models such as deep neural networks, the number of input data is often significantly smaller than the number of parameters to be estimated, so estimation based on classical statistics cannot be applied as is. Therefore, if sufficient learning data cannot be collected, it is difficult to estimate the generalization error, and it is difficult to estimate, for example, how much additional learning data is required in model learning. Furthermore, even if this estimation could be done mathematically, it would be difficult to proceed with the actual work of collecting additional training data unless the user could understand it. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Z. Liao et al., “Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, no. 1, pp. 15-26, 2020. [Non-Patent Document 2] Y. Takeishi, et al., “Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks” [Online], November 30, 2021, [Retrieved May 16, 2023], Internet<URL:https: / / arxiv.org / pdf / 2111.15256> [Non-Patent Document 3] Jacot et al., “Neural Tangent Kernel: Convergence and Generalization in Neural Networks” [Online], February 10, 2020, [Retrieved May 24, 2023], Internet<URL:https: / / arxiv.org / pdf / 1806.07572> [Non-Patent Document 4] Hidenori Takeshima, “Deep Learning and Its Application to Function Approximation for MR in Medicine: An Overview”, [Online], September 17, 2021, Magnetic Resonance in Medical Sciences, [Retrieved April 24, 2023], Internet<URL:https: / / doi.org / 10.2463 / mrms.rev.2021-0040> Summary of the Invention [Problem to be solved by the invention]

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to present information for supporting model training in a form that is easy for the user to understand, and to establish a mathematical basis for the information. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0005] The learning prediction device according to the present embodiment includes a calculation unit, a determination unit, and an output unit. The calculation unit calculates multidimensional data characterizing a machine learning model and associated data related to the multidimensional data. The determination unit generates support information for supporting a determination of a learning status of the machine learning model using a dataset based on whether the multidimensional data and the associated data meet a predetermined criterion. The output unit outputs the support information. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing a learning prediction device according to the present embodiment. [Diagram 2] FIG. 2 is a flowchart showing an example of the operation of the learning prediction device according to the present embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a graphical user interface (GUI) related to the operation and output of the learning prediction device according to this embodiment. [Figure 4] FIG. 4 is a flowchart showing details of the training process of the learning prediction device. [Diagram 5] FIG. 5 is a diagram showing a first example of an eigenvalue distribution according to the present embodiment. [Figure 6] FIG. 6 is a diagram showing a second example of an eigenvalue distribution according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, a learning prediction device, a learning prediction method, and a learning prediction program according to the present embodiment will be described with reference to the drawings. In the following embodiments, parts with the same reference numerals perform the same operations, and duplicated descriptions will be omitted as appropriate. Hereinafter, one embodiment will be described with reference to the drawings.

[0008] The learning prediction device according to this embodiment will be described with reference to the block diagram of FIG. The learning prediction device 1 according to this embodiment includes a processing circuit 10, a memory 11, an input interface 12, a communication interface 13, and a display 14.

[0009] The processing circuit 10 includes an acquisition function 101 , a training function 102 , a calculation function 103 , a judgment function 104 , and an output function 105 .

[0010] The acquisition function 101 acquires input information from a user. The acquisition function 101 also acquires a data set and a machine learning model. The data set is data that serves as training data, and is assumed to be, for example, medical images. Note that the data set is not limited to medical images, and may be other two-dimensional data, or one-dimensional data such as voice data or sensor data. The machine learning model may be any network structure used in the field of machine learning, such as a multilayer perceptron, a convolutional neural network, a neural network having a shortcut structure such as ResNet or DenseNet.

[0011] The training function 102 trains the machine learning model using the dataset and the loss function, and further calculates a training error for the dataset based on the loss function.

[0012] The calculation function 103 calculates multidimensional data that characterizes the machine learning model and associated data related to the multidimensional data.

[0013] The determination function 104 generates support information that supports the determination of the learning status of the machine learning model using the dataset based on whether the multidimensional data and the related data meet a predetermined criterion. The determination function 104 may also determine the learning status of the machine learning model using the dataset based on whether the multidimensional data and the related data meet a predetermined criterion.

[0014] The output function 105 outputs, for example, to the display 14, a graphical user interface (GUI) including support information such as information on the judgment result and information on whether training using the data set is sufficient.

[0015] The memory 11 stores training data including medical data such as medical images and test values, acquisition conditions, trained models, etc. The memory 11 is a semiconductor memory element such as a random access memory (RAM), a flash memory, a hard disk drive (HDD: Hard Disk Drive), a solid state drive (SSD: Solid State Drive), an optical disk, etc. The memory 11 may also be a drive device that reads and writes various information to and from a portable storage medium such as a CD-ROM drive, a DVD drive, or a flash memory.

[0016] The input interface 12 has a circuit for receiving various instructions and information input from a user. The input interface 12 has a circuit for an input device such as a pointing device such as a mouse or a keyboard. The circuits of the input interface 12 are not limited to circuits for physical operation parts such as a mouse and a keyboard. For example, the input interface 12 may have an electrical signal processing circuit for receiving an electrical signal corresponding to an input operation from an external input device provided separately from the learning prediction device 1 and outputting the received electrical signal to various circuits in the learning prediction device 1.

[0017] The communication interface 13 exchanges data with an external device via a wired or wireless connection. Since any common communication method can be used for the communication method and interface structure, a description thereof will be omitted here.

[0018] The display 14 displays a GUI (Graphical User Interface) for accepting various operations from the user. Any display can be used as appropriate, such as a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL display, an LED display, a plasma display, or a touch display that allows touch input. Note that the learning prediction device 1 may not include a display, and the GUI may be displayed on an external display, or the GUI may be displayed via a projector or the like.

[0019] Next, an example of the operation of the learning prediction device 1 according to this embodiment will be described with reference to the flowchart of FIG. In step SA1, the processing circuit 10 acquires input information related to the plug-in code input by a user to the GUI via, for example, the input interface 12, by the acquisition function 101. The plug-in code is a program code for executing the processing of the learning prediction device 1 according to the present embodiment in a framework related to a software library used to execute training of a machine learning model such as Python (registered trademark), PyTorch (registered trademark), Chainer (registered trademark), etc. The input information may be, for example, the type of framework, parameters in machine learning in the framework (such as the type of error function and optimizer, the number of epochs, etc.), parameters related to the execution environment, etc.

[0020] In step SA2, the processing circuit 10, for example by means of the calculation function 103, creates a plug-in code based on the input information obtained in step SA1. In step SA3, the processing circuit 10 trains the machine learning model using the training function 102, and calculates multidimensional data characterizing the machine learning model and its associated data. Specific examples of the multidimensional data and the associated data will be described later. In addition, the processing circuit 10 generates support information related to the trained machine learning model using the determination function 104.

[0021] In step SA4, the processing circuit 10 outputs the processing results, such as the multidimensional data and related data calculated in step SA3, and support information related to the judgment of the generalization performance of the machine learning model, by the output function 105. The multidimensional data, related data, and judgment results are assumed to be displayed on a GUI displayed on the display 14, but may be output to an external device such as an external monitor or a computer connected to a network via the communication interface 13. Alternatively, they may be output as sound via a speaker (not shown) or the like.

[0022] Next, an example of a GUI for a user to operate the learning prediction device 1 according to the present embodiment will be described with reference to FIG. The procedure for using the learning prediction device 1 includes inserting a plug-in code (step 1), conducting training and outputting the results using the plug-in code (step 2), and reading and analyzing the results (step 3).

[0023] Step 1 corresponds to step SA1 and step SA2 shown in FIG. 2, and inserting plug-in code is executed. In FIG. 3, a pull-down menu 31 for selecting a framework is displayed on the GUI. The user is prompted to select a framework from the pull-down menu 31. Here, "PyTorch" is selected, and the processing circuit 10 acquires the selection result from the user by the acquisition function 101. After that, when the plug-in code creation button 32 is pressed, plug-in code matching the framework is output as text data or a file. The user manually inserts the output plug-in code. An automatic creation mode, "Auto", may be prepared as the pull-down menu 31. When "Auto" is selected and the plug-in code creation button 32 is pressed, a file selection screen (not shown) is displayed. On the file selection screen, source code files (e.g., source code written in Python language) are displayed. When source code is selected, the source code is analyzed to search for a framework and a code insertion portion (e.g., a loss function calculation portion), and the plug-in code is automatically inserted. The pull-down menu 31 may be omitted, and the system may basically operate in the automatic creation mode using "Auto". In this case, only the plug-in code creation button 32 may be prepared.

[0024] 2, the above-mentioned multidimensional data and related data are calculated using the plug-in code prepared in step 1, and data related to the multidimensional data and related data is stored, for example, in the memory 11. The calculated multidimensional data and related data may be output to and stored in an external storage device (for example, HDD, network storage).

[0025] In step 3, which corresponds to step SA4 shown in Fig. 2, the user presses result read button 33, whereby the multidimensional data and related data stored in memory 11 or the like are read. For example, a file selection screen may be displayed to allow the user to select the file to be read. The read multidimensional data and related data are analyzed, and the analysis results are displayed on the screen. In the example of Figure 3, the multidimensional data is assumed to be an eigenvalue set for a matrix data set that characterizes, for example, the rate of change of a loss function used in training a machine learning model. Also, the related data is assumed to be boundary values ​​for the eigenvalue set. As support information, a graph 34 of the eigenvalue distribution and text 35 showing the analysis results are displayed. Specifically, the graph 34 and text 35 presented on the GUI display a correspondence graph between eigenvalues ​​sorted in descending order from the largest eigenvalue and eigenvalue index as graph 34. As text 35, the number of eigenvalues, boundary value α N Eigenvalue index, boundary value α N The value of and the judgment result of whether the number of learning samples is sufficient or not (in the example of FIG. 3, "Number of samples sufficient / insufficient?: sufficient") are displayed. Note that, without being limited to this, information such as a button for whether or not to execute re-learning, and the number of predictive learning samples required to reach a predetermined accuracy may be displayed on the GUI as support information.

[0026] Next, details of the training process in step SA2 of the learning prediction device 1 according to this embodiment will be described with reference to the flowchart in Fig. 4. Hereinafter, a case will be described in which the multidimensional data is an eigenvalue set and the related data is a boundary value.

[0027] In step SB1, the processing circuit 10 acquires training data by the acquisition function 101. The training data is input data x t and the correct answer data y t A set of t data items (t is an integer equal to or greater than 2) is a set of data items. The data set is also called a data set.

[0028] In step SB2, the processing circuit 10 trains the machine learning model using the data set by the training function 102, and calculates a training error based on a loss function. The machine learning model is assumed to be a deep neural network. In this embodiment, the loss function is assumed to be the mean squared error (MSE). The model is v (x) then the machine learning model f v Training error Er based on the loss function of (x) N(v) can be expressed as equation (1).

[0029]

number

[0030] where f v (x t ) is the machine learning model f v is the predicted value of (x). v is a random variable. N is the number of data points and is an integer equal to or greater than 2.

[0031] In step SB3, the processing circuit 10, using the calculation function 103, calculates eigenvalues ​​for a matrix that characterizes the rate of change of a loss function that obtains a training error, and calculates a set of eigenvalues ​​sorted in descending order. The reason for generating the eigenvalue set is as follows. First, according to Non-Patent Document 2, when the weight parameters of the final layer of a two-layer neural network are to be learned, the number of parameters to be learned does not depend much on the total number of parameters, but is determined by the input dimension and the number of data. Furthermore, the parameters to be learned are derived from a Fisher information matrix, which is an example of an index that defines the behavior of an estimator. The Fisher information matrix is ​​generally a positive definite matrix. In particular, when the dimension of the weight parameters of the final layer of a two-layer neural network is large, the Fisher information matrix becomes a square matrix corresponding to the dimension, and a large bias occurs in the eigenvalue distribution. In other words, even if the total number of parameters is huge, it can be said that the actual number of parameters to be trained (the number of execution parameters) is considerably smaller than the total number of parameters.

[0032] We will consider the case of a machine learning model that extends a two-layer neural network to include multiple intermediate layers, as in commonly used machine learning models. In this case, it is known that the Fisher information matrix of a general machine learning model asymptotically approaches a block diagonal matrix with blocks corresponding to each node. For this reason, it is predicted that bias will occur in the eigenvalue distribution even in a general machine learning model.

[0033] The machine learning model f according to this embodiment v In updating the parameters in the training process of (x), we assume that stochastic gradient descent (SGD) is used. In SGD, the direction of parameter updates is determined based on the gradient of the loss function. The gradient of the loss function in SGD can be understood from the quadratic gradient of the loss function, but calculating the quadratic gradient is not easy considering the computational cost. Therefore, the quadratic gradient of the training error is understood from the eigenvalues ​​of the Fisher information matrix. However, since the Fisher information matrix becomes large in general machine learning with a large number of parameters, the eigenvalues ​​of the Fisher information matrix can be calculated by approximately substituting the empirical Fisher information matrix J^ as an example of a matrix that characterizes the rate of change of the loss function with respect to the training error. Here, "^" indicates a superscript hat. Here, the Fisher information matrix is ​​assumed as the matrix, but any matrix that can characterize the quadratic gradient of the loss function, i.e., the rate of change of the loss function, may be used.

[0034] Furthermore, the empirical Fisher information matrix J^ is the training error Er N 1 / (2σ of the Hessian matrix of (v) 2 ) times (σ 2 is the variance of the observation noise. This can be seen from the following definition:

[0035] First, the machine learning model f v Suppose (x) is sufficiently complex that the true function f(x) can be expressed as equation (2).

[0036]

number

[0037] Here, when the input is x, the output y follows equation (3).

[0038]

number

[0039] In addition, the machine learning model f v Likelihood function p of (x) v (y N |x N ) satisfies equation (4).

[0040]

number

[0041] C is a constant. Here, the empirical Fisher information matrix J^ N is the left side of equation (4), that is, (-1 / N)logp v (y N |x N ) can be expressed as the Hessian matrix of the training error Er N 1 / (2σ of the Hessian matrix of (v) 2 ) times the empirical Fisher information matrix J^ N It can be said to be equivalent to

[0042] Furthermore, the empirical Fisher information matrix J^ N The expected value of can be defined as the Fisher information matrix J, and when N is large, the empirical Fisher information matrix J^ N can be approximated by the Fisher information matrix J(J^ N ≒J). Specifically, the empirical Fisher information matrix J^ N When trying to calculate the eigenvalues ​​of N 1 / (2σ of the Hessian matrix of (v) 2 ) times, but the observation noise σ 2 is not known, so the observation noise σ 2 An estimate of is required.

[0043] If we assume that the so-called Neural Tangent Kernel (NTK) theory shown in Non-Patent Document 3 holds, then the training error Er N (v) asymptotically approaches a quadratic function of v, and the approximation given by equation (5) holds.

[0044]

number

[0045] where v̂ is the optimal solution to the training error minimization problem. In addition, the estimated value v when early termination is performed during model training ~ (" ~ " indicates a superscript tilde), the training error value usually stagnates in the area where overfitting occurs, so it is considered that an approximation is valid, similar to equation (5). Therefore, the training error Er N (v * ) can be obtained as an approximation.

[0046]

number

[0047] Regarding the first term on the right hand side of equation (6), if we consider the trace, it can be expressed as equation (7).

[0048]

number

[0049] Also, N>α N If we assume that, we obtain equation (8). Note that α N is a boundary value of the eigenvalues, and corresponds to the number of eigenvalues ​​that satisfy a certain condition. N This will be described later in step SB4.

[0050]

number

[0051] Therefore, based on equations (6) and (8), the observation noise σ 2 can be estimated as shown in equation (9).

[0052]

number

[0053] Note that since "α N / N" on the right side becomes smaller as the number of data N increases, assuming the training data in general machine learning, it can be assumed that "α N << N". That is, the training error Er N (v ~ ) obtained by early termination in the training of the model can be estimated as the observation noise σ 2 .

[0054] As a result, the processing circuit 10 by the calculation function 103 can generate a set of eigenvalues of the empirical Fisher information matrix by rearranging the eigenvalues of the Hessian matrix of the training error Er 2 (v) using the observation noise σ N in descending order. As described above, since the empirical Fisher information matrix J^ N can be approximated by the Fisher information matrix J when the number of data is large, it is considered that a bias occurs in the set of eigenvalues.

[0055] In step SB4, the processing circuit 10 by the calculation function 103 calculates the boundary value α N of the eigenvalues. The predetermined condition for determining the boundary value α N may be calculated by, for example, the following method. Assuming as a premise that the absolute value of the true value of each parameter is 1 and the initial value of the parameter is determined by a standard normal distribution. When the empirical Fisher information matrix J N ^ can be approximated by the Fisher information matrix J, the standard deviation of the estimated value in the eigenvector direction of the eigenvalue λ of the Fisher information matrix J is 1 / (√(Nλ)). When 1 / (√(Nλ)) exceeds the range of the parameter, that is, when the standard deviation is greater than 1, it is not necessary to estimate the eigenvector direction. Therefore, from 1≧(1 / √(Nλ)), it is only necessary to estimate the eigenvalue λ that satisfies λ≧(1 / N). The processing circuit 10 by the calculation function 103 refers to the set of eigenvalues arranged in descending order of eigenvalues, and the number of eigenvalues λ that satisfy λ≧(1 / N) is the boundary value α NOn the other hand, for parameters related to eigenvalues ​​where λ<(1 / N), due to the nature of the backpropagation method (gradient method), it is estimated that the values ​​will hardly change from the initial values ​​when the number of iterations is small. In other words, by appropriately executing early termination in the training of a machine learning model, it is thought that the training of parameters related to these eigenvalues ​​will end with almost the initial values.

[0056] In step SB5, the processing circuit 10 judges the learning status based on the data set by using the judgment function 104, depending on whether or not one or more eigenvalues ​​having a value equal to or greater than the threshold value in the eigenvalue set meet a predetermined criterion. N It is determined whether or not one or more of the eigenvalues ​​satisfy a predetermined criterion. The predetermined criterion is, for example, a boundary value α N If the number of eigenvalues ​​is equal to or greater than the predetermined number, it is determined that the predetermined criterion is met, and the process proceeds to step SB6. On the other hand, if the number of eigenvalues ​​is less than the predetermined number, it is determined that the predetermined criterion is not met, and the process proceeds to step SB7.

[0057] In step SB6, the processing circuit 10 determines, via the decision function 104, that the eigenvalues ​​satisfy a predetermined criterion and that the number of data in the data set is sufficient, i.e., that the training data is sufficient.

[0058] In step SB7, the processing circuit 10 determines, by the determination function 104, that the eigenvalue does not meet a predetermined criterion and the number of data in the data set is insufficient, i.e., the training data is insufficient. As described above, by the processes of steps SB5 to SB7, the processing circuit 10 can determine the generalization performance of the machine learning model trained by the data set by the determination function 104.

[0059] In addition, the processing circuit 10 may output the eigenvalue set calculated in step SB3 and the number of data (whether the training data determined in steps SB6 and SB7 is sufficient or insufficient) as support information to an output device such as a display 14 in a GUI as shown in FIG. 3. Furthermore, when the processing circuit 10 determines that the training data is insufficient using the output function 105, a message prompting the acquisition of additional data may be displayed. In this case, the processing circuit 10 may automatically acquire additional data to be training data using the acquisition function 101. This allows the user to easily grasp the training status of the machine learning model.

[0060] In addition, based on the eigenvalue set, it is also possible to predict how the generalization error of the machine learning model will change when the number of training data items is increased. In other words, it is possible to determine whether sufficient training data has been collected so that the machine learning model being trained can achieve the desired learning performance.

[0061] Specifically, the processing circuit 10 uses the judgment function 104 to judge whether the eigenvalue distribution calculated in step SB3 is a N The generalization error is estimated based on the above. If we assume that the number of parameters in each layer of the machine learning model is equal, the generalization error can be estimated, for example, by equation (10).

[0062]

number

[0063] N is the number of training data, and D is the total number of parameters of the machine learning model. i indicates the i-th eigenvalue, where 1≦i≦N. The left side of equation (10) is the generalization error based on the eigenvalue, and "α N / 2N" is the boundary value α N The value corresponding to the eigenvalue larger than i≧αN λ i " is the boundary value α Nis the sum of eigenvalues ​​smaller than . On the other hand, the right side of the inequality sign is an eigenvalue that does not satisfy λ ≥ (1 / N). If the eigenvalue is small, the variance will be large and the error will be unreliable. However, when early termination of training of a machine learning model is adopted as described above, it is assumed that the eigenvalues ​​will remain almost at their initial values, so it is sufficient to focus on the left side of equation (10).

[0064] To predict the generalization error when the number of training data is increased using the above formula (10), λ as a function of i (1≦i≦D) is i needs to be estimated from the data. This is done by finding the eigenvalues ​​of the empirical Fisher information matrix. If the number of data is N, there are only N nonzero eigenvalues, so λ can only be found in the range i≦N. i Therefore, in this case, it is sufficient to assume a function form and predict the generalization error by extrapolation.

[0065] For example, if the amount of training data is increased and the fluctuation in generalization error is estimated, and a prediction is made that the generalization error will further decrease, it can be predicted that the learning performance of the machine learning model will improve by adding more training data, and it can be determined that sufficient training data has not been collected.

[0066] Next, a first example of an eigenvalue distribution according to this embodiment will be described with reference to FIG. FIG. 5 is a graph showing an eigenvalue set calculated using a data set consisting of a predetermined number of data, as an eigenvalue distribution. The horizontal axis is the eigenvalue index, and the vertical axis is a logarithmic scale of the magnitude of the eigenvalue. The eigenvalue distribution 51 is sorted in descending order of eigenvalues, and indexes are assigned starting from the largest eigenvalue. As shown in FIG. 5, the largest eigenvalue with the first index and the eigenvalue with the 100th index have eigenvalue magnitudes that differ by four or more orders of magnitude, and the eigenvalue values ​​fluctuate rapidly. It can therefore be seen that a bias occurs in the eigenvalue distribution of the empirical Fisher information matrix. In addition, the boundary value α N The area to the left of α NIn the region of the eigenvalue distribution with eigenvalues ​​of 1 or more, classical statistical estimation is applicable, so the central limit theorem holds and the generalization error can be estimated since it follows a normal distribution. On the other hand, the boundary value α N The area to the right of α N A region of the eigenvalue distribution having an eigenvalue less than 1 is a region in which the eigenvalues ​​hardly move from their initial values.

[0067] Next, a second example of the eigenvalue distribution according to this embodiment will be described with reference to FIG. FIG. 6 shows the eigenvalue distribution in the same way as FIG. 5, but shows multiple eigenvalue distributions calculated for multiple data sets with different numbers of data. Here, the cases where the numbers of data included in the data sets are 90, 180, 360, 720, 1080, 1440, and 1800 are shown. In addition, the boundary value α N The value used is calculated for a data set with N being 1,800 samples.

[0068] As shown in Figure 6, when comparing the eigenvalue distributions of data sets with data numbers N = 90, 180, and 360, the shapes of the eigenvalue distributions vary. On the other hand, when comparing the eigenvalue distributions of data numbers N = 1440 and 1800, the shapes of the eigenvalue distributions based on eigenvalues ​​with smaller index numbers are almost the same. In other words, it can be seen that the shape converges to a certain shape as the number of data N increases. Therefore, if the processing circuit 10 calculates the eigenvalue distribution by changing the number of data in the dataset using the calculation function 103, and determines using the judgment function 104 that the shape of the eigenvalue distribution has converged to a certain shape, it can be determined that the desired generalization performance has been obtained and that the number of data in the dataset, which is training data, is sufficient.

[0069] From another perspective, the eigenvalue distribution 51 for a data set of 360 samples is compared with the eigenvalue distribution 61 for a data set of 720 samples. N There exists an eigenvalue for which the number of samples is equal to or larger than the boundary value α N has not yet reached that level. That is, the data set from which the eigenvalue distribution 61 was obtained has a sufficient number of samples, and the machine learning model is sufficiently trained. On the other hand, the data set from which the eigenvalue distribution 51 was obtained has an insufficient amount of training data, and the machine learning model is not sufficiently trained. In other words, from the data set at hand, N It is predicted that by selecting a number of datasets exceeding 100,000, and training a machine learning model, it is possible to obtain the same generalization performance with a smaller number of samples in the dataset. This can also reduce the time and memory costs of training the model. Of course, the processing circuit 10 may use the output function 105 to display on a GUI the change in eigenvalue distribution resulting from differences in the number of data items included in the data set.

[0070] According to the theorem described in Non-Patent Document 2, when limited to learning the parameters of the final layer of a deep neural network, when predicting the generalization error, if the number of data N is increased, the boundary value α N It can be seen that there is a range in which the boundary value α increases rapidly. This can be estimated from the shape of the eigenvalue distribution. In other words, in this case, the shape of the eigenvalue distribution becomes stepped in the direction in which the index increases, and several groups of eigenvalues ​​within a certain range are formed. At this time, it can be said that there is a tendency for the eigenvalues ​​to suddenly become smaller at a certain index, and the number of eigenvalues ​​with roughly the same value to suddenly increase. This can be considered a special case of a phenomenon called a plateau, which is empirically known in deep learning. In other words, even in general deep neural networks, the boundary value α N This suggests that there may be a range in which the rate of change increases rapidly.

[0071] In addition, the user may make the judgment, and for example, the processing circuit 10 may determine the convergence of the shape of the eigenvalue distribution by the judgment function 104, and determine the boundary value α at which the eigenvalue may become next smallest. N And the number of data may be determined.

[0072] According to the present embodiment described above, multidimensional data characterizing a machine learning model and associated data related to the multidimensional data are calculated, and support information for supporting the determination of the learning status of the machine learning model using a dataset is generated based on whether the multidimensional data and the associated data conform to a predetermined criterion, and the support information is output. This allows a user to refer to the support information to support the training of the model. In addition, a machine learning model is trained using a dataset, an eigenvalue set for the dataset of a matrix that characterizes the rate of change of a loss function as multidimensional data is calculated, and the learning status based on the dataset is determined based on whether one or more eigenvalues ​​in the eigenvalue set that have a value equal to or greater than a boundary value conform to a predetermined criterion. For example, the suitability of the number of data in the dataset for obtaining a desired generalization error performance for a trained model is determined. If the one or more eigenvalues ​​conform to the predetermined criterion, it can be determined that the number of training data is sufficient. In addition, by estimating the generalization error when the number of training data is increased, it can be determined whether additional training data needs to be collected. As shown in this embodiment, since the empirical Fisher matrix can be calculated based on the Hessian matrix of the training error to estimate the parameters of the machine learning model, eigenvalues ​​can be calculated using a matrix with dimensions corresponding to the number of training data. As a result, it is possible to support model training by predicting the generalization performance of the model and the amount of data required for training while significantly saving computational resources.

[0073] The various functions of the processing circuit 10 may be stored in the memory 11 in the form of a program executable by a computer. In this case, the processing circuit 10 can be said to be a processor that realizes the functions corresponding to each program by reading the programs corresponding to these various functions from the memory 11 and executing them. In other words, the processing circuit 10 in a state in which each program has been read out has the multiple functions and the like shown in the processing circuit 10 in FIG. 1.

[0074] 1, a single processor constitutes the processing circuit 10 and various functions are realized, but multiple independent processors may be combined to constitute the processing circuit 10, and each processor may execute a program to realize a function. In other words, each of the above-mentioned functions may be configured as a program, and one processing circuit may execute each program, or a dedicated independent program execution circuit may execute a specific function.

[0075] In addition, each function according to the embodiment can be realized by installing a program for executing the above-mentioned processes in a computer such as a workstation and expanding the program in memory. In this case, the program for causing the computer to execute the above-mentioned method can be stored in a storage medium such as a magnetic disk (such as a hard disk), an optical disk (such as a CD-ROM or DVD), or a semiconductor memory and distributed.

[0076] In addition, the term "processor" used in the above description refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)).

[0077] Although some embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0078] 1 Learning prediction device 10 Processing circuit 11. Memory 12 Input Interface 13 Communication Interface 14 Display 31 Pull-down Menu 32 Plugin code creation button 33 Result Load Button 34 Graphs 35 Text 51,61 Eigenvalue distribution 101 Acquisition Function 102 Training Function 103 Calculation Function 104 Judgment function 105 Output Function

Claims

1. A calculation unit that calculates multidimensional data that characterizes a machine learning model and associated data related to the multidimensional data; a determination unit that generates support information for supporting a determination of a learning status of the machine learning model using a dataset based on whether the multidimensional data and the related data conform to a predetermined criterion; and an output unit that outputs the support information; A learning prediction device comprising:

2. The learning prediction device according to claim 1 , wherein the output unit displays the multidimensional data and the support information on a GUI (Graphical User Interface).

3. The learning prediction device according to claim 1 , further comprising an acquisition unit that acquires, from a user, a selection result of a framework type for executing training of the machine learning model.

4. the output unit displays a GUI (Graphical User Interface) on which a button for creating a plug-in code for the framework is arranged; The acquisition unit acquires a creation instruction from a user when the creation button is pressed, The learning prediction device according to claim 3 , wherein the calculation unit generates the plug-in code related to training the machine learning model in response to obtaining the generation instruction.

5. A training unit that trains a machine learning model using the dataset and the loss function; A calculation unit that calculates multidimensional data that characterizes the machine learning model and associated data related to the multidimensional data; A determination unit that determines a learning status of the machine learning model using the dataset based on whether the multidimensional data and the related data meet a predetermined criterion; A learning prediction device comprising:

6. The multidimensional data is a set of eigenvalues ​​for the dataset of a matrix characterizing a rate of change of a loss function used in training the machine learning model; The learning prediction device according to claim 1 , wherein the associated data is a boundary value for the eigenvalue set.

7. The learning prediction device according to claim 6 , wherein the determination unit determines that training using the data set is sufficient when one or more eigenvalues ​​in the eigenvalue set that have a value equal to or greater than the boundary value meet the predetermined criterion.

8. The learning prediction device according to claim 6 , wherein the determination unit determines that training using the data set is insufficient when the one or more eigenvalues ​​having a value equal to or greater than the boundary value do not conform to the predetermined criterion.

9. The learning prediction device according to claim 6 , wherein the predetermined criterion is that the number of the one or more eigenvalues ​​having a value equal to or greater than the boundary value is equal to or greater than a threshold value.

10. The learning prediction device according to claim 8 , further comprising an output unit that outputs, when it is determined that training using the data set is insufficient, a message indicating that additional data acquisition is necessary.

11. The learning prediction device according to claim 6 , wherein the determination unit assumes a functional form from an eigenvalue distribution based on the eigenvalue set, and predicts a generalization error of the machine learning model by extrapolation.

12. The learning prediction device according to claim 6 , wherein the determination unit determines a degree of convergence of a shape of an eigenvalue distribution based on the eigenvalue set, and predicts a generalization error of the machine learning model.

13. Calculating multi-dimensional data characterizing the machine learning model and associated data related to the multi-dimensional data; generating support information for supporting a determination of a learning status of the machine learning model using a dataset based on whether the multidimensional data and the related data conform to a predetermined criterion; outputting the support information; ,Learning prediction methods.

14. Train a machine learning model using the dataset and the loss function; Calculating multi-dimensional data characterizing the machine learning model and associated data related to the multi-dimensional data; A learning prediction method, which determines the learning status of the machine learning model using the dataset based on whether the multidimensional data and the related data meet predetermined criteria.

15. On the computer, A calculation function that calculates multidimensional data that characterizes a machine learning model and associated data related to the multidimensional data; A determination function that generates support information that supports a determination of a learning status of the machine learning model using a dataset based on whether the multidimensional data and the related data meet a predetermined criterion; and an output function for outputting the support information; A learning prediction program to achieve this.

16. On the computer, A training function that trains a machine learning model using the dataset and the loss function; A calculation function that calculates multidimensional data that characterizes the machine learning model and associated data related to the multidimensional data; A determination function that determines a learning status of the machine learning model using the dataset based on whether the multidimensional data and the related data meet a predetermined criterion; A learning prediction program to achieve this.

Images