Neural network training device, neural network training method, and program

Abstract

The present invention provides a neural network training technique in which a weight parameter is not restricted to have non-negative values while monotonicity is still guaranteed. This neural network training device trains a neural network, which includes: an encoder that converts an input vector into a latent variable vector having latent variables as the elements; and a decoder that converts the latent variable vector into an output vector, such that the input vector and the output vector are approximately the same. The training is performed such that the output vector has monotonicity with respect to the latent variable vector.

Description

Neural network learning device, neural network learning method, and program

[0001] The present invention relates to techniques for training neural networks.

[0002] Various methods have been proposed for analyzing large amounts of high-dimensional data. For example, there are methods that use Non-negative Matrix Factorization (NMF) described in Non-Patent Document 1 and Infinite Relational Model (IRM) described in Non-Patent Document 2. These methods make it possible to discover characteristic properties of data and to group data with common properties into clusters.

[0003] Lee, DD and Seung, HS, “Learning the parts of objects by non-negative matrix factorization,” Nature, 401, pp.788-791, 1999.Kemp, C., Tenenbaum, JB, Griffiths, TL, Yamada, T. and Ueda, N., “Learning systems of concepts with an infinite relational model,” AAAI06(Proceedings of the 21st national conference on Artificial intelligence, pp.381-388, 2006.

[0004] Analysis methods using NMF and IRM often require the advanced analytical skills of data analysts. However, data analysts often lack detailed knowledge of the high-dimensional data being analyzed (hereafter referred to as "target data"), and in such cases, collaboration with a data expert is necessary. However, this collaboration sometimes does not go smoothly. Therefore, a method is needed that allows analysis by data experts alone, without the need for a data analyst.

[0005] Consider analysis using a neural network including an encoder and a decoder, such as the variational autoencoder (VAE) described in Non-Patent Document 1. Here, the encoder is a neural network that converts an input vector into a latent variable vector, and the decoder is a neural network that converts the latent variable vector into an output vector. The latent variable vector is a vector with lower dimensions than the input vector and the output vector, and is a vector with latent variables as elements. Converting high-dimensional data to be analyzed using an encoder trained to make the input and output vectors approximately identical can compress the data into low-dimensional secondary data. However, because the relationship between the data to be analyzed and the secondary data is unknown, this cannot be applied to analysis work as is. Here, "training to make the data approximately identical" refers to ideally training to make them completely identical, but in reality, due to training time constraints and other factors, training must be performed to make the data approximately identical. Therefore, training is performed in a manner that the data are deemed identical and the processing is terminated when a predetermined condition is met.

[0006] (Reference Non-Patent Document 1: Kingma, D. P. and Welling, M., "Auto-encoding variational bayes," arXiv preprint arXiv:1312.6114, 2013.) Reference Patent Document 1 provides a technology for training a neural network including an encoder and a decoder so that the larger the magnitude of a certain property included in an input vector, the larger a certain latent variable included in a latent variable vector becomes. The neural network training device trains a neural network including an encoder that converts an input vector into a latent variable vector and a decoder that converts the latent variable vector into an output vector so that the input vector and the output vector become approximately identical, and the training is performed in a manner that satisfies the conditions that all weight parameters of the decoder are non-negative and non-positive values. (Reference Patent Document 1: International Publication No. WO 2022 / 244050) However, the constraint that all weight parameters be non-negative values ​​guarantees that the output vector has monotonicity (described later) with respect to the latent variable vector, but weight parameters that satisfy monotonicity also exist other than non-negative values, and for example, if a weight parameter becomes too close to 0, a problem may arise in which learning does not progress. Therefore, an object of the present invention is to provide a neural network learning technology that does not restrict weight parameters to non-negative values ​​and that guarantees monotonicity.

[0007] One aspect of the present invention is a neural network training device that trains a neural network including an encoder that converts an input vector into a latent variable vector having latent variables as elements, and a decoder that converts the latent variable vector into an output vector, so that the input vector and the output vector become substantially identical. The training is performed by regarding two latent variable vectors as a first latent variable vector and a second latent variable vector, and when, for at least one element of the latent variable vector, the element value of the first latent variable vector is greater than the element value of the second latent variable vector, and for all remaining elements of the latent variable vector, the element value of the first latent variable vector is equal to or greater than the element value of the second latent variable vector, and the output vector obtained by converting the first latent variable vector is referred to as a first output vector, and the output vector obtained by converting the second latent variable vector is referred to as a second output vector, so that, for at least one element of the output vector, the element value of the first output vector is greater than the element value of the second output vector, and for all remaining elements of the output vector, the element value of the first output vector is equal to or greater than the element value of the second output vector.

[0008] According to the present invention, it is possible to train a neural network including an encoder and a decoder while the weight parameters are not restricted to non-negative values ​​and while maintaining monotonicity.

[0009] FIG. 1 is a diagram showing an example of data to be analyzed. FIG. 2 is a diagram for explaining constraints when learning weight parameters of a decoder. FIG. 3 is a block diagram showing the configuration of a neural network learning device 100. FIG. 4 is a flowchart showing the operation of the neural network learning device 100. FIG. 5 is a diagram showing an example of the functional configuration of a computer that realizes each device in an embodiment of the present invention.

[0010] Hereinafter, embodiments of the present invention will be described in detail. Components having the same functions are given the same numbers, and duplicated explanations will be omitted.

[0011] Before describing each embodiment, the notation used in this specification will be explained.

[0012] ^ (caret) represents a superscript, e.g., x y^z Yesz is a superscript to x, and x y^z Yes z is a subscript to x. Also, _ (underscore) represents a subscript. For example, x y_z Yes z is a superscript to x, and x y_z Yes z is a subscript to x.

[0013] In addition, the superscripts "^" and "~" such as ^x and ~x for a certain letter x should be written directly above the "x", but due to restrictions on the notation in the specification, they are written as ^x and ~x.

[0014] <Technical Background> This section describes a training method for a neural network including an encoder and a decoder used in an embodiment of the present invention. The neural network used in an embodiment of the present invention is a neural network including an encoder that converts an input vector into a latent variable vector and a decoder that converts the latent variable vector into an output vector. In an embodiment of the present invention, this neural network is trained so that the input vector and the output vector become approximately identical. In an embodiment of the present invention, the latent variable is trained as having the following feature (hereinafter referred to as feature 1) so that the larger a certain latent variable included in the latent variable vector is, the greater the magnitude of a certain property included in the output vector will be.

[0015] [Feature 1] Learning is performed so that the output vector has monotonicity with respect to the latent variable vector. Here, the output vector having monotonicity with respect to the latent variable vector means that there is a monotonically increasing relationship in which the output vector increases as the latent variable vector increases. Note that the magnitude of the output vector and latent variable vector is based on the order relationship with respect to the vector (i.e., a relationship defined using the order relationship with respect to each element of the vector), and for example, the following order relationship can be used.

[0016] Vector v=(v1, …, v n ), v'=(v'1, …, v'n ), v≦v' is true for all elements of vectors v and v', that is, the i-th element v of vector v i , the i-th element v' of vector v' i (where i=1, …, n), v i ≦v' i This means that the following holds true.

[0017] Specifically, learning to make the output vector monotonic with respect to the latent variable vector means learning to make the latent variable vector have the following first relationship with the output vector.

[0018] The first relationship is a relationship in which, when two latent variable vectors are designated as a first latent variable vector and a second latent variable vector, the element value of the first latent variable vector is greater than the element value of the second latent variable vector for at least one element of the latent variable vector, and the element values ​​of the first latent variable vector are equal to or greater than the element values ​​of the second latent variable vector for all remaining elements of the latent variable vector, the output vector obtained by transforming the first latent variable vector is designated as a first output vector, and the output vector obtained by transforming the second latent variable vector is designated as a second output vector, and the element value of the first output vector is greater than the element value of the second output vector for at least one element of the output vector, and the element values ​​of the first output vector are equal to or greater than the element values ​​of the second output vector for all remaining elements of the output vector.

[0019] For convenience, when expressing the first relationship, an expression that the output vector has a monotonically increasing relationship with the latent variable vector may be used.

[0020] By training the latent variable vector to have the above-mentioned feature 1, a latent variable vector is provided that satisfies the condition that the larger the magnitude of a certain latent variable included in the latent variable vector, the larger the magnitude of a certain property included in the output vector.

[0021] In the embodiment of the present invention, the latent variable may be learned as having the following feature (hereinafter referred to as feature 2) in addition to the above feature 1.

[0022] [Feature 2] The system learns so that the values ​​that latent variables can take are easy for users to understand.

[0023] By learning the latent variable vector to have the above-mentioned feature 2 in addition to the above-mentioned feature 1, a latent variable vector that satisfies the condition that the larger the magnitude of a certain latent variable included in the latent variable vector, the larger the magnitude of a certain property included in the output vector will be provided as a parameter that is easy for general users to understand.

[0024] We will now explain the constraints for training a neural network including an encoder that outputs a latent variable vector having the above feature 1. Specifically, we will explain the following two constraints.

[0025] [Constraint 1] Learn to minimize the loss function.

[0026] [Constraint 2] The decoder weight parameters are trained to have monotonicity.

[0027] First, we will explain the neural network to be trained. For example, the following VAE can be used. The encoder and decoder are each two-layer neural networks, with the first and second layers of the encoder and the first and second layers of the decoder fully connected. The input vector that is input to the first layer of the encoder is, for example, a 60-dimensional vector. The output vector that is output from the second layer of the decoder is a vector that reconstructs the input vector. Furthermore, a sigmoid function is used as the activation function for the second layer of the encoder. This ensures that the values ​​of the elements of the latent variable vector (i.e., each latent variable) that are output from the encoder are between 0 and 1. Note that the latent variable vector is a vector with fewer dimensions than the input vector, for example, a five-dimensional vector. For example, Adam (see Non-Patent Document 2) can be used as a training method.

[0028] (Reference non-patent document 2: Kingma, DP and Jimmy B., “Adam: A Method for Stochastic Optimization,” arXiv:1412.6980, 2014)

[0029] Next, we will explain the loss function including the loss term of Constraint 1. The loss function L is a term L that ensures that the output vector has monotonicity with respect to the latent variable vector. mono For example, the loss function L can be defined as the function defined by the following equation. Note that the term L in the following equation mono In order to provide an efficient explanation, the formula includes a term related to feature 2 in addition to a term related to feature 1, and the term related to feature 2 will be explained as appropriate.

[0030] term L RC , L prior are terms related to the reconstruction error and Kullback-Leibler divergence, respectively, which are commonly used in VAE training. For example, the term L RC is the binary cross entropy (BCE) of the error between the input vector and the output vector, and the term L prior is the Kullback-Leibler divergence between the distribution of the latent variables output by the encoder and the prior distribution. Figure 1 shows a matrix that represents the correctness of students' answers to test questions, with a correct answer being 1 and an incorrect answer being 0. The rows represent a list of correct answers by all students for each question, and the columns represent a list of correct answers by each student for all questions. Here, Q1, ..., Q60 in Figure 1 represent the first question, ..., 60th question, and N1, ..., N S represents the first student, ..., Sth student. In this case, each column is an input vector that is input to the encoder, and S is the number of training data. Since each element of the input vector is either 1 or 0, the prior distribution for the example in Figure 1 can be, for example, a mean μ = 0.5 and a variance σ 2 A Gaussian distribution with σ = 1 can be used.

[0031] term L mono is the second term L syn-encoder (p) , L syn-decoder (p) The term L syn-encoder (p) and the term Lsyn-decoder (p) is a term related to feature 2.

[0032] term L syn-encoder (p) is a term related to artificial data that is the upper limit of the range of values ​​that all elements of the input vector can take, or the lower limit of the range of values ​​that all elements of the input vector can take. For example, in the example shown in Figure 1, where each element of the input vector is either 1 or 0, the term L syn-encoder (p) is a term related to artificial data in which the input vector is a vector (1, ..., 1) corresponding to all correct answers, or a vector (0, ..., 0) corresponding to all incorrect answers. Specifically, the term L syn-encoder (1) is the binary cross entropy between the latent variable vector that is the output of the encoder when the input vector is the vector (1, …, 1) corresponding to all correct answers, and the ideal latent variable vector (1, …, 1) where all elements are 1 (i.e., the upper limit of the range of possible values) when the input vector is the vector (1, …, 1) corresponding to all correct answers. Also, the term L syn-encoder (2) is the binary cross-entropy between the latent variable vector that is the output of the encoder when the input vector is the vector (0, …, 0) corresponding to all incorrect answers, and the ideal latent variable vector (0, …, 0) with all elements 0 (i.e., the lower limit of the range of possible values) when the input vector is the vector (0, …, 0) corresponding to all incorrect answers. The term L syn-encoder (1) is based on the requirement that when the input vector is (1, ..., 1), that is, when all elements of the input vector are 1 (i.e., the upper limit of the range of possible values), all elements of the latent variable vector should be 1 (i.e., the upper limit of the range of possible values), and the term L syn-encoder (2)is based on the requirement that when the input vector is (0, ..., 0), that is, when all elements of the input vector are 0 (i.e., the lower limit of the range of possible values), it is desirable that all elements of the latent variable vector be 0 (i.e., the lower limit of the range of possible values).

[0033] On the other hand, the term L syn-decoder (p) is a term related to artificial data that is the upper limit of the range of values ​​that all elements of the output vector can take, or the lower limit of the range of values ​​that all elements of the output vector can take. For example, in the example of Figure 1, where each element of the input vector has a value of either 1 or 0, the term L syn-decoder (p) is a term related to artificial data in which the output vector is a vector (1, ..., 1) corresponding to all correct answers, or an artificial data in which the output vector is a vector (0, ..., 0) corresponding to all incorrect answers. Specifically, the term L syn-decoder (1) is the binary cross entropy between the output vector, which is the decoder output when the latent variable vector is a vector (1, …, 1) where the values ​​of all elements are the upper limit of the range of possible values, and the ideal output vector, which is a vector (1, …, 1) where all elements are 1 (i.e., corresponding to all correct answers) when the values ​​of all elements of the latent variable vector are the upper limit of the range of possible values. Also, the term L syn-decoder (2) is the binary cross-entropy between the output vector, which is the decoder output when the latent variable vector is a vector (0, …, 0) whose values ​​are the lower limit of the range of possible values ​​for all elements, and the ideal output vector, which is a vector (0, …, 0) whose elements are all 0 (i.e., corresponding to all incorrect answers) when the values ​​of all elements of the latent variable vector are the lower limit of the range of possible values. The term L syn-decoder (1)is based on the requirement that when the latent variable vector is (1, ..., 1), that is, when all elements of the latent variable vector are 1 (i.e., the upper limit of the range of possible values), all elements of the output vector should be 1 (i.e., the upper limit of the range of possible values), and the term L syn-decoder (2) is based on the requirement that when the latent variable vector is (0, ..., 0), that is, when all elements of the latent variable vector are 0 (i.e., the lower limit of the range of possible values), it is desirable that all elements of the output vector be 0 (i.e., the lower limit of the range of possible values).

[0034] L syn-encoder (p) , L syn-decoder (p) The loss function L also contains the term L mono By including this in the loss function L, the neural network is trained so that the values ​​of all elements of the latent variable vector are within the range [0, 1] (i.e., the range of possible values).

[0035] Next, we will explain the learning method for Constraint 2. In the explanation of the learning method for Constraint 2, the number of the input vector used for learning is s (s is an integer between 1 and S, and S is the number of learning data), the number of the element of the latent variable vector is j (j is an integer between 1 and J, and k is an integer between 1 and K, and K is an integer greater than J), and the number of the element of the input vector and the output vector is k (k is an integer between 1 and K, and K is an integer greater than J). s and the input vector X s The latent variable vector obtained by transforming s and the latent variable vector Z s The output vector obtained by converting s and the input vector X s The kth element of x sk and the output vector P s The kth element of p sk and the latent variable vector Z s The jth element of z sj Let's say.

[0036] The encoder receives an input vector X sis the latent variable vector Z s Any function that converts the input data into a vector can be used, for example, a general VAE encoder. The loss function used for learning does not need to be a special one, and can be a conventionally used loss function, for example, the above-mentioned term L RC and the term L prior The sum of these can be used as the loss function.

[0037] The decoder generates a latent variable vector Z s The output vector P s and is learned under certain constraints.

[0038] We will explain the constraints of a decoder using an example of a decoder consisting of one layer. Let us consider an input vector X1, X2, ..., X, where the student's answers to a test question with K questions are represented by 1 for correct answers and 0 for incorrect answers. S If , the input vector of the sth student is X s =(x s1 , x s2 , ..., x sK ) and the input vector X s The latent variable vector obtained by converting s =(z s1 , z s2 , ..., z sJ ) and the latent variable vector Z s The output vector obtained by converting s =(p s1 , p s2 , ..., p sK ) is the probability that a student will correctly answer each test question, and therefore will need to have various categories of ability, such as writing ability and diagramming ability, each with its own weight. In order to make each element of the latent variable vector correspond to each category of ability, and to make the value of the latent variable corresponding to that category larger the greater the ability of each category that a student possesses, the probability p sk the j-th latent variable z sjThe weight w for the kth test problem is given as jk It is recommended to express this as equation (3) with a non-negative value.

[0039] where σ is the sigmoid function and b k is the bias term for the kth problem. The bias term b k is a term corresponding to the difficulty of the k-th problem that does not depend on the ability of each category mentioned above. In other words, in the case of a decoder consisting of one layer, all weights w jk (j=1, …, J, k=1, …, K) are constrained to be non-negative, and the training input vector X s is the latent variable vector Z s and the latent variable vector Z s The output vector P s and a neural network containing a decoder that converts the input vector X s and the output vector P s and are approximately identical, an encoder can be obtained that obtains a latent variable vector for each category of ability from an input vector, which is a vector representing the student's answers to each test question, with correct answers being 1 and incorrect answers being 0, such that the greater the ability in a certain category, the larger a certain latent variable becomes.

[0040] To train a neural network that guarantees monotonicity, the latent variable z sj a small non-negative value ε j When adding the weight w, it is guaranteed that the following equation is satisfied. jk (j=1, …, J, k=1, …, K). For simplicity, if the number of latent variables J is 2, The weight w that satisfies 1k , w 2k (k=1, ..., K), where ε1 and ε2 are non-negative values. Here, if we perform a first-order approximation using Maclaurin expansion, we get Further, if we perform a first-order approximation using Maclaurin expansion, we get In other words, the w that satisfies this 1k , w 2k However, since this is a first-order approximation, equation (7) is only approximately satisfied. Using a second-order approximation or similar would improve the accuracy of the approximation, but would result in the equation becoming more complex.

[0041] In Reference Patent Document 1, in order to satisfy Equation (4), all weight parameters of the decoder are non-negative values ​​(w 1k >0, w 2k >0) and training was performed.

[0042] On the other hand, the solution space of equation (7) is area A in Figure 2. In other words, area A satisfies the following equation. In other words, to satisfy monotonicity, w 1k , w 2k There is also a possibility that either of these is negative.

[0043] On the other hand, the w used in Patent Document 1 1k >0, w 2k > 0 is in region B, which is included in region A, so monotonicity is guaranteed, but it can be seen that it is part of the solution space that satisfies monotonicity. Therefore, in Reference Patent Document 1, there is a possibility that learning will not progress if the latent variable approaches 0 too much.

[0044] Therefore, taking these into consideration, if the number of latent variables is J, the following conditions must be satisfied to ensure monotonicity. ε is set in advance (see the embodiment for a setting example). The cost function for this purpose can be as follows: Here, α is a weight for the monotonicity cost that is set in advance, and a value of 0 means that this cost is not used. When log has a negative value, its value rapidly decreases toward -∞, which allows learning to satisfy equation (10).

[0045] Now, depending on the value of α, for example, w 1k and w 2k The ranges of the latent variables w jk However, it may not be a parameter that is easy for general users to understand. By adding β, it can be made into an easy-to-understand parameter. β is a weight that is set in advance. Here, w jk Although the first norm of is used, the second norm may also be used.

[0046] As mentioned above, in the example shown in Figure 1, each column represents a list of correct answers for each student. Using a trained encoder, the 60-dimensional list of correct answers for each student is converted into five-dimensional secondary data. Because the trained encoder's conversion ensures that the latent variable vector is monotonic with respect to the input vector, this five-dimensional compressed secondary data reflects the characteristics of the list of correct answers for each student. For example, if a list of correct answers for a student's Japanese or arithmetic test is converted to a latent variable vector, the elements of the secondary data, which is the latent variable vector, can be data corresponding to, for example, writing ability or diagramming ability. Therefore, by analyzing secondary data instead of the list of correct answers for each student, the burden on the analyst can be reduced.

[0047] First Embodiment A neural network training device 100 uses training data to train parameters of a neural network to be trained. Here, the neural network to be trained includes an encoder that converts an input vector into a latent variable vector and a decoder that converts the latent variable vector into an output vector. The latent variable vector is a vector with lower dimensions than the input vector and the output vector, and is a vector with latent variables as elements. Furthermore, the parameters of the neural network include weight parameters and bias parameters of the encoder and weight parameters and bias parameters of the decoder. Training is performed so that the input vector and the output vector are substantially identical. Furthermore, training is performed so that the output vector is monotonic with respect to the latent variable vector.

[0048] Here, it is described that the possible values of the elements of the input vector and the output vector are either 1 or 0, and the range of possible values of the latent variables, which are the elements of the latent variable vector, is [0, 1]. Note that the fact that the possible values of the elements of the input vector and the output vector are either 1 or 0 is just an example. The range of possible values of the elements of the input vector and the output vector may be [0, 1], or further, the range of possible values of the elements of the input vector and the output vector may not be [0, 1]. That is, letting a and b be any numbers satisfying a < b, the range of possible values of the elements of the input vector and the range of possible values of the elements of the output vector can be set as [a, b].

[0049] Hereinafter, the neural network learning device 100 will be described with reference to FIGS. 3 to 4. FIG. 3 is a block diagram showing the configuration of the neural network learning device 100. FIG. 4 is a flowchart showing the operation of the neural network learning device 100. As shown in FIG. 3, the neural network learning device 100 includes an initialization unit 110, a learning unit 120, an end condition determination unit 130, and a recording unit 190. The recording unit 190 is a component configured to appropriately record information necessary for the processing of the neural network learning device 100. The recording unit 190, for example, records initialization data used for the initialization of the neural network. Here, the initialization data refers to the initial values of the parameters of the neural network, for example, the initial values of the weight parameters and bias parameters of the encoder, and the initial values of the weight parameters and bias parameters of the decoder.

[0050] The operation of the neural network learning device 100 will be described according to FIG. 4.

[0051] In S110, the initialization unit 110 performs the initialization process of the neural network using the initialization data. Specifically, the initialization unit 110 sets initial values for each parameter of the neural network. Also, α in Equation (10), β in Equation (11), and ε in Equation (4) etc. j are set in advance. For example, let α = 0.01, β = 0.01, etc., and ε jSet all of (j = 1, 2, …, J) to 1.

[0052] In S120, the learning unit 120 takes the learning data as input and performs a process of updating each parameter of the neural network using the learning data (hereinafter referred to as the parameter update process), and outputs the parameters of the neural network together with the information necessary for the end condition determination unit 130 to determine the end condition (for example, the value of the loss function, the number of times the parameter update process has been performed, etc.). The learning unit 120 uses the loss function to learn the neural network, for example, by the error backpropagation method. That is, in each parameter update process, the learning unit 120 performs a process of updating each parameter of the encoder and the decoder so that the loss function becomes smaller. Here, the loss function is as follows. L = L RC + L prior + L mono + L const + L norm

[0053] In S130, the end condition determination unit 130 takes the parameters of the neural network output in S120 and the information necessary for determining the end condition as input, and determines whether the end condition, which is a condition regarding the end of learning, is satisfied (for example, the amount of change in the value of the loss function due to the update is below the threshold, the number of times the parameter update process has been performed has reached a predetermined number of repetitions, etc.). If the end condition is satisfied, the encoder parameters obtained in the last S120 are output as learned parameters and the process ends. On the other hand, if the end condition is not satisfied, the process returns to the process of S120.

[0054] (Variant example) Instead of setting the range of possible values of the latent variables, which are elements of the latent variable vector, to [0, 1], it may be set to [m, M] (where m < M), or as described above, the range of possible values of the elements of the input vector and the output vector may be set to [a, b]. Furthermore, the range of possible values may be set individually for each element of the latent variable vector, or the range of possible values may be set individually for each element of the input vector and the output vector.

[0055] In the above explanation, an example has been described in which the number of dimensions of the latent variable vector is two or more, but the number of dimensions of the latent variable vector may also be one. That is, the above-mentioned J may also be 1. When the number of dimensions of the latent variable vector is one, the above-mentioned "latent variable vector" can be read as "latent variable", and "the value of at least one element of the latent variable vector" can be read as "the value of the latent variable", and there is no condition regarding "all remaining elements of the latent variable vector".

[0056] Finally, we will explain the analysis process. Using an encoder with trained parameters (trained encoder), the data to be analyzed is converted into lower-dimensional secondary data. Here, the secondary data refers to the latent variable vector obtained by inputting the data to be analyzed into the trained encoder. Because this secondary data is lower-dimensional than the data to be analyzed, analyzing the secondary data is easier than directly analyzing the data to be analyzed.

[0057] According to the first embodiment, it is possible to train a neural network including an encoder and a decoder so as to obtain decoder parameters such that the magnitude of a certain property included in an output vector increases as a certain latent variable included in a latent variable vector increases. Then, by using the trained encoder to convert high-dimensional analysis target data and obtaining low-dimensional secondary data as the analysis target, it is possible to reduce the burden on the analyst.

[0058] Furthermore, a device (terminal) for using the device, system, or method of the present invention via a network (telecommunications line) may also be provided. The "device (terminal) for use" may be provided with functions (e.g., control function, decoding function, restoration function, input / output function, etc.) necessary to obtain the effects of implementing the device, system, or method of the present invention.

[0059] 5 is a diagram showing an example of the functional configuration of a computer that realizes each of the above-mentioned devices (i.e., each node). The processing in each of the above-mentioned devices can be implemented by having the recording unit 2020 load a program for causing the computer to function as each of the above-mentioned devices, and having the control unit 2010, input unit 2030, output unit 2040, etc. operate.

[0060] The device of the present invention may, for example, be a single hardware entity that includes an input unit to which a keyboard or the like can be connected, an output unit to which an LCD display or the like can be connected, a communication unit to which a communication device (e.g., a communication cable) capable of communicating with an external device can be connected, a CPU (which may also include a central processing unit, cache memory, registers, etc.), RAM and ROM as memory, an external storage device such as a hard disk, and buses connecting these input unit, output unit, communication unit, CPU, RAM, ROM, and external storage device so that data can be exchanged between them. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing to a recording medium such as a CD-ROM. An example of a physical entity equipped with such hardware resources is a general-purpose computer.

[0061] The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required for processing these programs (the programs may be stored in a ROM, which is a read-only storage device, for example, instead of an external storage device). Data obtained by processing these programs is stored in RAM, the external storage device, etc. as appropriate.

[0062] In a hardware entity, each program stored in an external storage device (or ROM, etc.) and the data required to process each program are loaded into memory as needed, and interpreted, executed, and processed by the CPU as appropriate, resulting in the CPU realizing a predetermined function (each component represented as the above, ... unit, ... means, etc.).

[0063] The present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. Furthermore, the processes described in the above embodiments are not limited to being executed in chronological order according to the order described, but may be executed in parallel or individually depending on the processing capacity of the device that executes the processes or as needed.

[0064] As described above, when the processing functions of the hardware entities (apparatuses of the present invention) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entities should have are described by a program. Then, by executing this program on a computer, the processing functions of the hardware entities are realized on the computer.

[0065] The program describing the processing contents can be recorded on a non-transitory computer-readable recording medium. Examples of computer-readable recording media include magnetic recording devices, optical disks, magneto-optical recording media, and semiconductor memories. Specifically, examples of magnetic recording devices include hard disk drives, flexible disks, and magnetic tapes; optical disks include DVDs (Digital Versatile Discs), DVD-RAMs (Random Access Memory), CD-ROMs (Compact Disc Read Only Memory), and CD-Rs (Recordable / Rewritable) / RWs; magneto-optical recording media include MOs (Magneto-Optical Discs), and semiconductor memories include EEP-ROMs (Electronically Erasable and Programmable-Read Only Memory).

[0066] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0067] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored in its storage device and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with each program transferred from the server computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. In this embodiment, the program includes information used for processing by a computer that is equivalent to a program (e.g., data that is not a direct instruction to the computer but has properties that define computer processing).

[0068] In addition, in this embodiment, a hardware entity is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0069] The foregoing description of the embodiments of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Modifications and variations are possible in light of the above teachings. The embodiments were chosen and described to provide the best illustration of the principles of the invention and to enable those skilled in the art to utilize the invention in various embodiments and with various modifications as may be suitable for the practical uses contemplated. All such modifications and variations are within the scope of the invention as defined by the appended claims, interpreted in accordance with the breadth to which they are fairly, legally, and equitably entitled.

Claims

1. A neural network training device that trains a neural network including an encoder that converts an input vector into a latent variable vector whose elements are latent variables, and a decoder that converts the latent variable vector into an output vector, so that the input vector and the output vector become substantially identical, wherein the training is performed by: defining two latent variable vectors as a first latent variable vector and a second latent variable vector; and when, for at least one element of the latent variable vector, the value of the element of the first latent variable vector is greater than the value of the element of the second latent variable vector, and for all remaining elements of the latent variable vector, defining an output vector obtained by converting the first latent variable vector as a first output vector and an output vector obtained by converting the second latent variable vector as a second output vector, such that, for at least one element of the output vector, the value of the element of the first output vector is greater than the value of the element of the second output vector, and for all remaining elements of the output vector, the value of the element of the first output vector is greater than the value of the element of the second output vector, and 2. The neural network learning device of claim 1, wherein the number of latent variables is J, and ε j Let be a non-negative value, and w jk are the weight parameters of the decoder, and training is performed to satisfy the following condition: Neural network learning device.

3. A neural network training method for training a neural network including an encoder that converts an input vector into a latent variable vector having latent variables as elements, and a decoder that converts the latent variable vector into an output vector, so that the input vector and the output vector become substantially identical, wherein the training is performed by: defining two latent variable vectors as a first latent variable vector and a second latent variable vector; and when, for at least one element of the latent variable vector, the value of the element of the first latent variable vector is greater than the value of the element of the second latent variable vector, and for all remaining elements of the latent variable vector, defining an output vector obtained by converting the first latent variable vector as a first output vector and an output vector obtained by converting the second latent variable vector as a second output vector, such that, for at least one element of the output vector, the value of the element of the first output vector is greater than the value of the element of the second output vector, and for all remaining elements of the output vector, the value of the element of the first output vector is greater than the value of the element of the second output vector, and 4. A program for causing a computer to function as the neural network learning device according to claim 1.

Images