Optimization device, optimization method, and program

Abstract

One aspect of the present invention is an optimization device provided with a control unit that executes a candidate estimation model that is a mathematical model for estimating, in accordance with a target word, which is an assigned word, candidates for a word to be inputted into a result generation model, which is a mathematical model that outputs a result corresponding to the inputted word. The candidate estimation model is a mathematical model for solving, on the basis of the target word and word vector information indicating a correspondence relationship where one or a plurality of words and vector(s) in a prescribed vector space are associated in a one-to-one manner, the optimization problem of minimizing a loss value that includes: a quantity that represents the difference between a target vector representing the target word and a linear combination of non-target vectors representing words different from the target word, the quantity being such that said difference becomes smaller as the quantity becomes smaller; and a quantity that indicates the number of non-target vectors having a non-zero weighting in the linear combination, the quantity being such that said number becomes smaller as the quantity becomes smaller.

Description

Optimization device, optimization method, and program

[0001] The present invention relates to an optimization device, an optimization method, and a program.

[0002] There is a mathematical model that, when a word, sentence, or other term is input, outputs data in a predetermined format, such as image data, sound data, text data, or chemical compound data, in accordance with the content of the input term.

[0003] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in Neural Information Processing Systems 33 (2020): 6840-6851. Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[0004] Such mathematical models change their output results only in response to the input words, so if a user wants to change the output result, they have no choice but to change the input words. However, controlling the output result with words is not always easy. For example, if a user wants to output a result that shows almost the opposite content, they can simply input an antonym, but if they want to change the output result only slightly, they can simply input a word that is similar in meaning, but it is not always easy for the user to find such a word. As such, using a mathematical model that outputs results in response to input words can be a heavy burden for the user.

[0005] In view of the above circumstances, an object of the present invention is to provide a technique that reduces the effort required to use a mathematical model that outputs a result according to an input word.

[0006] One aspect of the present invention is an optimization device that includes a control unit that executes a candidate estimation model, which is a mathematical model that estimates word candidates to be input to a result generation model, which is a mathematical model that outputs a result according to an input word, according to a target word that is a given word, and the candidate estimation model is a mathematical model that solves an optimization problem to obtain weights for the linear combination that minimizes a loss value including: a quantity that represents the difference between a linear combination of a target vector, which is a vector representing the target word, and non-target vectors, which are vectors representing other words different from the target word, the smaller the quantity, and a quantity that represents the number of non-target vectors whose weight in the linear combination is not zero, the smaller the quantity, based on word vector information that indicates a pre-established one-to-one correspondence between one or more pre-given words and vectors in a predetermined vector space, and the target word.

[0007] One aspect of the present invention is an optimization method comprising: a control step of executing a candidate estimation model, which is a mathematical model that estimates word candidates to be input to a result generation model, which is a mathematical model that outputs a result according to an input word, according to a target word that is a given word; the candidate estimation model is a mathematical model that solves an optimization problem of obtaining weights for the linear combination that minimizes a loss value including: a quantity that represents a difference between a linear combination of a target vector that is a vector representing the target word and non-target vectors that are vectors representing other words different from the target word, the smaller the quantity, and a quantity that represents the number of non-target vectors whose weight in the linear combination is not zero, the smaller the quantity, based on word vector information that indicates a pre-established one-to-one correspondence between one or more pre-given words and vectors in a predetermined vector space and the target word.

[0008] One aspect of the present invention is a program for causing a computer to function as the optimization device described above.

[0009] The present invention makes it possible to reduce the effort required to use a mathematical model that outputs results according to input words.

[0010] An explanatory diagram for explaining an optimization device of an embodiment. A diagram showing an example of the hardware configuration of the optimization device of an embodiment. A diagram showing an example of the flow of processing executed by the optimization device of an embodiment. A diagram showing an example of the flow of processing executed by an optimization device of a modified example.

[0011] 1 is an explanatory diagram illustrating an optimization device 1 according to an embodiment. The optimization device 1 reduces the effort required by a user of a result generation model to find words to input to the result generation model. More specifically, it reduces the effort required by the user to find other words that are similar in meaning to a word that the user has thought of.

[0012] The result generation model is a mathematical model that accepts input of words and outputs a result according to the input words. Hereinafter, how the optimization device 1 reduces the user's effort will be described.

[0013] The optimization device 1 includes a control unit 11 having a processor 91 such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Network Processing Unit) and a memory 92 connected via a bus.

[0014] The control unit 11 executes, for example, a candidate estimation model. By executing the candidate estimation model, the optimization device 1 can reduce the effort required by the user to find other words that are similar in meaning to the word the user is thinking of (hereinafter referred to as "user effort"). The control unit 11 may execute a dictionary optimization model. By executing a dictionary optimization model in addition to the candidate estimation model, the optimization device 1 can further reduce the user effort. The control unit 11 may execute a dictionary optimization model without executing a candidate estimation model. Even in such a case, the optimization device 1 can reduce the user effort.

[0015] The dictionary optimization model will be explained in a modified example. First, the candidate estimation model will be explained.

[0016] The candidate prediction model is a mathematical model that predicts word candidates to be input to the result generation model according to a given word (hereinafter referred to as a "target word"). That is, the candidate prediction model is a mathematical model that accepts input of a target word and predicts word candidates to be input to the result generation model according to the input target word. An example of a target word is the above-mentioned word imagined by the user.

[0017] More specifically, the candidate prediction model is a mathematical model that obtains word candidates to be input to the result generation model by solving a word optimization problem. The word optimization problem is an optimization problem that obtains variable values ​​that minimize word loss based on word vector information and target words input to the candidate prediction model. Hereinafter, the process of solving the word optimization problem will be referred to as word optimization.

[0018] Word vector information is information indicating a one-to-one correspondence established in advance between one or more given words and vectors in a predetermined vector space (hereinafter referred to as "word vector space"). Note that the smaller the difference between vectors in the word vector space, the closer the meanings of the words associated with the vectors. Therefore, a word vector space is a vector space that has vectors (hereinafter referred to as "word vectors") that are associated with words in advance, and that are associated with a single word but not with multiple words, and the smaller the difference between word vectors, the closer the meanings of the words associated with the word vectors.

[0019] Two vectors with opposite directions have opposite meanings for the associated words, such as "beautiful" and "ugly." The difference between vectors is, for example, the distance between the vectors. If the vectors are unit vectors, the difference between the vectors may be the difference from 1, which is the dot product of the vectors.

[0020] The word vector space is, for example, the CLIP embedding space.

[0021] The word loss includes a first approximation loss and a word count loss. The first approximation loss is a quantity representing the difference between a target vector and a linear combination of non-target vectors (hereinafter referred to as the "first linear combination"), and the smaller the value, the smaller the difference between the target vector and the first linear combination.

[0022] A target vector is a vector in a word vector space that represents a target word. That is, a target vector is a word vector associated with the target word. A non-target vector is a vector in a word vector space that represents a word other than the target word. That is, a non-target vector is a word vector associated with a word other than the target word.

[0023] The word count loss is a quantity indicating the number of non-target vectors whose weights in the first linear combination are not zero, and the smaller the quantity, the fewer the number of non-target vectors whose weights in the first linear combination are.

[0024] <More Specific Example> To make it easier to understand the concepts presented so far, a more specific example is given. Suppose a target vector y is divided into four non-target vectors v １ , v ２ , v ３ and v ４ By w １ v １ +w ２ v ２ +w ３ v ３ +w ４ v ４ In the case where it is approximated as １ ~w ４ Of which lol １ and w ３ The indication of word loss when only the weight w １ ~w ４ Of which lol ３ is smaller than the value indicated by the word loss when only

[0025] Weight w １is the number of non-target vectors v in the first linear combination that approximates the target vector V. １ The weight of w ２ is the number of non-target vectors v in the first linear combination that approximates the target vector y. ２ The weight of w ３ is the number of non-target vectors v in the first linear combination that approximates the target vector y. ３ The weight of w ４ is the number of non-target vectors v in the first linear combination that approximates the target vector y. ４ is the weight.

[0026] For example, if the target vector y is associated with the word "car," then the non-target vector v １ ~v ４ are respectively associated with the words "wheel", "body", "engine", and "steering wheel", for example.

[0027] To facilitate understanding, consider the weight in the first linear combination when there is a non-target vector associated with the word "tomato" and the target vector y is "car." "Tomato" is less related to "car" than "wheel," "body," "engine," or "steering wheel." Therefore, when the target vector y is "car," the weight of the non-target vector associated with "tomato" in the first linear combination is likely to be closer to zero than the weights for "wheel," "body," "engine," or "steering wheel."

[0028] For ease of understanding, we will also explain the first approximation loss in more detail. １ , v ２ , v ３ and v ４ By w １ v １ +w ２ v ２ +w ３ v ３ +w ４ v ４is approximated, the first approximation loss is the target vector y and the first linear combination w １ v １ +w ２ v ２ +w ３ v ３ +w ４ v ４ This shows the difference between

[0029] <Regarding the optimization variables in word optimization, the meaning of weights, and the fact that word optimization is estimation using a candidate prediction model> We will now explain the optimization variables in word optimization, the meaning of weights, and the fact that word optimization is estimation using a candidate prediction model.

[0030] In word optimization, the weights of the first linear combination are treated as optimization variables. Therefore, the optimization results in the weights of each term of the first linear combination. Naturally, the weights indicate how much the non-target vectors acting on them contribute to the linear combination.

[0031] For example, if the weight of one term is 0.9, the weight of another term is 0.1, and the weights of the remaining terms are all zero, this means that the target vector is represented by a non-target vector with a weight of 0.9 and a non-target vector with a weight of 0.1.

[0032] This means that the target vector is a word vector that is closer to the non-target vector with a weight of 0.9 than to other word vectors in the first linear combination. As described above, since close word vectors are associated with words with similar meanings, this means that the meaning of the word associated with the non-target vector with a weight of 0.9 is closer to the meaning of the target word.

[0033] In this way, the weight represents how close the meaning of the word associated with the non-target vector to the meaning of the target word is, and the larger the weight, the closer the meaning of the word associated with the non-target vector to the meaning of the target word.

[0034] Therefore, the meaning of a word associated with a non-target vector with a weight of 0.1 is more different from the meaning of the target word than the meaning of a word associated with a non-target vector with a weight of 0.9. On the other hand, the meaning of a word associated with a non-target vector with a weight of 0.1 is less different from the meaning of the target word than the meaning of a word associated with a non-target vector with a weight of zero. This is the meaning represented by the weight.

[0035] So far, we have explained weights as the semantic closeness of the target word to the semantic of the non-target word to which the weight is applied. In other words, the larger the weight, the higher the probability that the word associated with the non-target vector to which the weight is applied is a candidate word to be input to the result generation model. Therefore, obtaining weights through word optimization means estimating candidate words to be input to the result generation model. Therefore, obtaining weights through word optimization means obtaining candidate words to be input to the result generation model instead of the target word.

[0036] Therefore, the candidate estimation model obtains weight values ​​for each non-target vector of the first linear combination by solving the word optimization problem, and obtains words associated with the non-target vectors on which the weights act that satisfy the candidate establishment condition as candidates for words to be input to the result generation model. The candidate establishment condition is a predetermined condition regarding the determination of candidates using the weights in the first linear combination.

[0037] The candidate validity condition is, for example, that the weight value obtained by word optimization is greater than a predetermined value. The candidate validity condition may also be, for example, that the weight value obtained by word optimization is within the Nth largest value (N is a predetermined integer of 1 or greater).

[0038] <Effects of Word Optimization> The effects of word optimization will now be described. First, the content represented by the first approximation loss and the effect of optimization that reduces the value of the first approximation loss will be described, and then the content represented by the word count loss and the effect of optimization that reduces the value of the word count loss will be described, thereby explaining the effects of word optimization.

[0039] <What the First Approximation Loss Represents and Its Effects> The following describes what the first approximation loss represents and the effects of optimization that reduces the value of the first approximation loss. Since the definition of the first approximation loss is as defined above, the first approximation loss represents how accurately a target word can be approximated using other words, with a smaller value indicating a higher degree of approximation. Since close word vectors in the word vector space represent corresponding words that are close in meaning, the higher the degree of approximation, the more words that are closer in meaning to the target word are estimated by the candidate estimation model. This is what the first approximation loss represents.

[0040] As described above, in a word optimization problem, optimization is performed so as to reduce the value of losses including the first approximation loss. Therefore, in a word optimization problem, optimization is performed so as to reduce the value of the first approximation loss. A smaller value of the first approximation loss means a higher degree of approximation. Therefore, by performing optimization to reduce the first approximation loss, words with meanings closer to the meaning of the target word can be obtained with higher accuracy than if optimization is not performed. This is an example of the effect achieved by optimization to reduce the first approximation loss.

[0041] <What word count loss represents and its effects> We will now explain what word count loss represents and the effects of optimization to reduce the word count loss value. As described above, the weight in the first linear combination represents the probability that a word is a candidate for input to the result generation model. Therefore, a weight of zero means that the word is not a candidate for input to the result generation model. Therefore, a smaller word count loss value means that fewer word candidates will be input to the result generation model.

[0042] As mentioned above, in the word optimization problem, optimization is performed to reduce the loss value, including the word count loss. Therefore, in the word optimization problem, optimization is performed to reduce the word count loss value. Reducing the word count loss value means reducing the number of word candidates input to the result generation model, and the reason we try to reduce the number of candidates is because it is more desirable. Here, we will explain why this is desirable.

[0043] For simplicity, the following description will be given citing a specific usage situation of the optimization device 1. Here, as a specific situation, consider a situation in which there is a user who wants to find words to input to a result generation model by using the optimization device 1. Furthermore, in this situation, consider as specific examples a case in which the optimization device 1 estimates, for example, 10,000 words as candidates, and a case in which the optimization device 1 estimates 10 words as candidates.

[0044] In this case, a user who knows the words estimated by the optimization device 1 would naturally choose to select words to input to the result generation model from 10 candidate words rather than selecting words to input to the result generation model from 10,000 candidate words. This is because even with 10,000 candidates, there are too many options and the user has to work hard to make a selection. In this way, information overload also increases the user's work load.

[0045] Therefore, to prevent such information overload situations from occurring, optimization is performed to reduce the word loss value in the word optimization problem. This is an example of the effect of optimization to reduce the word loss value.

[0046] Since the effects of reducing the value of the first approximation loss and the value of the word count loss are as described above, word optimization that reduces the value of the word loss can reduce the effort required by a user to find words to be input to a result generation model in place of target words. Since the work of a user finding words to be input to a result generation model is part of the effort required to use the result generation model, it can be said that word optimization that reduces the value of the word loss can reduce the effort required to use a mathematical model that outputs results according to input words.

[0047] <Example of Word Loss Using a Formula> The word loss including the first proximity loss and the word count loss may be, for example, a linear sum of the first proximity loss and the word count loss. Hereinafter, the word loss expressed as a linear sum of the first proximity loss and the word count loss will be referred to as the first word loss. The first word loss is expressed, for example, by the following formula (1):

[0048]

[0049] y is the target vector. D is a matrix in which all non-target vectors in the word vector space are arranged in a predetermined direction, either row-wise or column-wise. The vector w is a vector in which each element represents the weight of one non-target vector in the first linear combination. Therefore, for example, if the target vector y is w １ v １ +w ２ v ２ +w ３ v ３ +w ４ v ４ If the vector w is approximated as １ , w ２ , w ３ , w ４ In such a case, the matrix D is, for example, D=(v １ v ２ v ３ v ４ )

[0050] The first term in Equation (1) represents the first approximation loss. R(w) represents the word count loss. Therefore, R is a function that takes a tensor such as a vector as an argument and outputs the number of non-zero elements among the elements of the argument. In the example of Equation (1), the argument is the vector w.

[0051] λ is a weighting coefficient that acts on the function R. The value of λ is a predetermined value.

[0052] The word loss may be, for example, a loss (hereinafter referred to as a "second word loss") obtained by further adding a constraint that the weights in the first linear combination are non-negative to the first word loss. The word optimization problem using the second word loss is solved under the condition that the weights in the first linear combination are non-negative. The second word loss is expressed, for example, by the following equation (2).

[0053]

[0054] Under the condition that the weights in the first linear combination can be negative, a word vector representing an antonym of the word associated with the target vector may be used to approximate the target vector, such as "ugly" for "beautiful."

[0055] A negative weight in the first linear combination indicates that the non-target vectors on which the weight acts have the opposite direction to the target vector. Reverse vector directions in the word vector space mean that the word vectors are associated with words that have opposite meanings. Therefore, negative weights are used when expressing the first linear combination using non-target vectors associated with antonyms of the target word, such as "ugly" for "beautiful."

[0056] Therefore, when the second word loss is used, the probability that a candidate word close to the target word will be predicted by the candidate prediction model is higher than when the first word loss is used.

[0057] The word loss may be, for example, a loss (hereinafter referred to as a "third word loss") obtained by adding a constraint to the second word loss that the sum of the weights in the first linear combination is 1. The word optimization problem using the third word loss is solved under the condition that the weights in the first linear combination are non-negative, as well as the further condition that the sum of the weights in the first linear combination is 1. The third word loss is expressed, for example, by the following equation (3).

[0058]

[0059] 1 Ｔ The equation w=1 indicates the condition that the sum of the weights in the first linear combination is 1. Ｔ w means the dot product of a vector whose elements are all 1 and the vector w. As mentioned above, weights can also be said to represent probabilities. Therefore, the condition that the sum of the weights in the first linear combination is 1 means that the probability is 1.

[0060] Normalization is often performed in mathematics and physics. A typical example of normalization would be a complete orthonormal system. In probability theory, the probability of all events is also normalized to 1. The trace of a density matrix is ​​normalized to 1. This normalization is performed because it makes mathematical processing easier. If mathematical processing becomes easier, the computational load will be reduced. The same applies to the condition that the sum of the weights in the first linear combination is 1, and the computational load will be reduced compared to when the sum of the weights in the first linear combination is not 1.

[0061] <Example of Optimization> In word optimization, the candidate prediction model may use any optimization method to obtain the variable values ​​that minimize the word loss value, as long as the variable values ​​are obtained based on the word vector information and the target words input to the candidate prediction model.

[0062] For example, if the word loss is a first word loss, the optimization method may be, for example, the proximal gradient method described in Reference 1. For example, if the word loss is a second word loss or a third word loss, the optimization method may be, for example, the alternating direction multiplier method described in Reference 2.

[0063] Reference 1: Passty, Gregory B. "Ergodic convergence to a zero of the sum of monotone operators in Hilbert space." Journal of Mathematical Analysis and Applications 72.2 (1979): 383-390.

[0064] Reference 2: Gabay, Daniel, and Bertrand Mercier. "A dual algorithm for the solution of nonlinear variational problems via finite element approximation." Computers & mathematics with applications 2.1 (1976): 17-40.

[0065] The optimization process when the word loss is the first word loss is, for example, a process of executing the calculation expressed by the following equation (4).

[0066]

[0067] k represents the iteration index. The parameter μ is the matrix D Ｔ It is a predetermined positive value equal to or less than the reciprocal of the largest eigenvalue of D, and represents the step size. prox is a proximity mapping, and is defined as in the following equation (5).

[0068] x represents the input of the proximity map. Note that the proximity map may be calculated using any known technique. For example, in the case of the L1 norm, the proximity map may be calculated using soft thresholding.

[0069] The optimization process when the word loss is the second word loss or the third word loss is a process of executing the optimization algorithm expressed by, for example, the following equation (6).

[0070]

[0071] The function proj is defined by the following equation (7) and represents a distance projection onto a convex set C, which will be described later.

[0072]

[0073] If the word loss is the second word loss, the distance projection may be obtained by, for example, non-negative clipping of the input vector. The input vector is specifically a weight vector. If the word loss is the third word loss, the distance projection may be obtained by, for example, projection onto a probability simplex (see Reference 3). The parameter γ is a predetermined positive value that represents the step size.

[0074] Reference 3: Wang, Weiran, and Miguel A. Carreira-Perpinan. "Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application." arXiv preprint arXiv:1309.1541 (2013).

[0075] The convex set C represents a constraint condition in the second word loss or the third word loss. Therefore, when the word loss is the second word loss, the convex set C is expressed by the following formula (8).

[0076]

[0077] Furthermore, when the word loss is the third word loss, the convex set C is expressed by the following formula (9).

[0078]

[0079] 2 is a diagram showing an example of the hardware configuration of the optimization device 1 according to an embodiment. The optimization device 1 includes a control unit 11 having a processor 91 and a memory 92 connected by a bus, and executes a program. By executing the program, the optimization device 1 functions as a device including the control unit 11, an interface unit 12, and a storage unit 13.

[0080] More specifically, the processor 91 reads out a program stored in the storage unit 13 and stores the read out program in the memory 92. The processor 91 executes the program stored in the memory 92, whereby the optimization device 1 functions as a device including the control unit 11, the interface unit 12, and the storage unit 13.

[0081] The control unit 11 controls the operation of each functional unit included in the optimization device 1. The control unit 11 acquires a target word, for example, via the interface unit 12. The control unit 11 executes, for example, a candidate estimation model. By executing the candidate estimation model, candidate words to be input to the result generation model are obtained.

[0082] The control unit 11 acquires the target word from the device that transmitted the target word, for example, via the interface unit 12. Note that the control unit 11 does not necessarily have to acquire the target word via the interface unit 12. The control unit 11 acquires, for example, information stored in the memory unit 13. Specifically, the process of acquiring the information stored in the memory unit 13 is reading.

[0083] The interface unit 12 includes a communication interface for connecting the optimization device 1 to an external device. The interface unit 12 communicates with the external device via wired or wireless communication. The external device is, for example, a device that is a source of the target word. In such a case, the interface unit 12 acquires the target word by communicating with the device that is a source of the target word.

[0084] The interface unit 12 may be configured to include input devices such as a mouse, keyboard, or touch panel. The interface unit 12 may be configured as an interface that connects these input devices to the optimization device 1. In this way, the interface unit 12 accepts input of various information to the optimization device 1 via the input device, either wired or wireless. Note that the various data acquired by the interface unit 12 described above does not necessarily have to be acquired by the interface unit 12 itself, and may be input to an input device.

[0085] The interface unit 12 outputs, for example, various types of information. The interface unit 12 includes a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The interface unit 12 may be configured as an interface that connects these display devices to the optimization device 1. The interface unit 12 outputs, for example, information input to the interface unit 12.

[0086] The storage unit 13 is configured using a computer-readable storage medium device (non-transitory computer-readable recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 13 stores various information related to the optimization device 1. The storage unit 13 stores, for example, word vector information in advance. The storage unit 13 stores, for example, various information generated by the operation of the control unit 11. The storage unit 13 stores, for example, information acquired by the interface unit 12.

[0087] 3 is a diagram showing an example of the flow of processing executed by the optimization device 1 in the embodiment. The control unit 11 acquires target words (step S101). The control unit 11 may acquire the target words, for example, via the interface unit 12, or by reading out target words stored in advance in the storage unit 13. Next, the control unit 11 executes a candidate estimation model (step S102). By executing the candidate estimation model, word optimization is performed and candidates for words to be input to the result generation model are estimated.

[0088] As described above, the optimization device 1 includes a control unit 11 that executes a candidate estimation model. Execution of the candidate estimation model performs word optimization, and predicts candidates for words to be input to the result generation model. Therefore, for the reasons described in <Effects of Word Optimization> above, the optimization device 1 can reduce the effort required by a user of a result generation model to find words to input to the result generation model. In other words, the optimization device 1 can reduce the effort required to use a mathematical model that outputs results according to input words.

[0089] (Variations) <Regarding the Dictionary Optimization Model> In the word optimization, a word vector space is used. As described above, the word vector space is a vector space having word vectors, and the smaller the difference between the word vectors, the closer the meanings of the words associated with the word vectors are.

[0090] Here, we will explain a situation where some subspaces in a vector space have a higher density of word vectors than other subspaces. Technologies for predicting words similar to a target word, including candidate prediction models and the technologies described in Non-Patent Documents 1 and 2, predict words similar to the target word using a word vector space such as a clip embedding space.

[0091] Although the details of how to use the word vector space vary depending on the technology, these techniques estimate words close to the target word by estimating word vectors close to the target vector. Therefore, if the word vectors are not sparsity, many word vectors associated with different meanings will exist in a small spatial region, increasing the possibility that words that are not close to the target word will be estimated.

[0092] Therefore, if the sparsity of word vectors in the word vector space increases, words close to the target word can be estimated with higher accuracy, thereby reducing the user's effort required to use a mathematical model that outputs results according to the input words.

[0093] Although the details will be described later, the above-mentioned dictionary optimization model is a technique for increasing the sparsity of word vectors in a word vector space. Therefore, by executing the dictionary optimization model, the control unit 11 reduces the user's effort required to use a mathematical model that outputs results according to input words.

[0094] The control unit 11 may execute both the candidate estimation model and the dictionary optimization model, or may execute the dictionary optimization model without executing the candidate estimation model. Even when executing the dictionary optimization model without executing the candidate estimation model, the word vector space obtained by the dictionary optimization model can be used by a device other than the optimization device 1 that executes the candidate estimation model or a device that executes a technology for estimating words similar to the target word other than the candidate estimation model.

[0095] Therefore, by implementing the dictionary optimization model, the user's effort in using a mathematical model that outputs results according to the words entered is reduced. The dictionary optimization model will now be described in more detail.

[0096] <More details of the dictionary optimization model> The dictionary optimization model is a mathematical model that solves a dictionary optimization problem and updates word information vectors based on the solution results. The dictionary optimization problem is an optimization problem that obtains the values ​​of variables that minimize the dictionary optimization loss, which is a loss that includes a first dictionary loss, a second dictionary loss, and an element count loss, plus a first dictionary constraint. Hereinafter, the process of solving the dictionary optimization problem will be referred to as dictionary optimization.

[0097] The first dictionary loss is a quantity representing the difference between the dictionary matrix and the first transformed dictionary matrix, and the smaller the quantity, the smaller the difference between the dictionary matrix and the first transformed dictionary matrix. The dictionary matrix is ​​a matrix in which each vector indicated by the word information vector is arranged in a predetermined direction, either row-wise or column-wise. Each vector indicated by the word information vector is a vector in the word vector space and associated with a word, and therefore is specifically a word vector. The first transformed dictionary matrix is ​​the result of performing an affine transformation on the dictionary matrix. Hereinafter, the affine transformation on the dictionary matrix will be referred to as the first dictionary transformation.

[0098] The content of the first dictionary transformation is the target of optimization in the dictionary optimization, and therefore the variables that minimize the dictionary optimization loss include each element of the matrix representing the first dictionary transformation.

[0099] The second dictionary loss is a quantity representing the difference between the first transformed dictionary matrix and the second transformed dictionary matrix, and the smaller the quantity, the smaller the difference between the first transformed dictionary matrix and the second transformed dictionary matrix. The second transformed dictionary matrix is ​​the result of performing a linear transformation on the first transformed dictionary matrix. Hereinafter, the linear transformation performed on the first transformed dictionary matrix will be referred to as the second dictionary transformation.

[0100] The content of the second dictionary transformation is the target of optimization in the dictionary optimization, and therefore the variables that minimize the dictionary optimization loss include each element of the matrix representing the second dictionary transformation.

[0101] The element number loss is a quantity that indicates the number of non-zero elements among the elements of the matrix representing the second dictionary transformation, and the smaller the quantity, the fewer the number of non-zero elements among the elements of the matrix representing the second dictionary transformation.

[0102] The first dictionary constraint is a condition that the diagonal elements of the matrix representing the second dictionary transformation are 0. The meaning of the first dictionary constraint will be described later.

[0103] In this way, in dictionary optimization, the contents of the first dictionary conversion and the contents of the second dictionary conversion are optimized so as to minimize the dictionary optimization loss, so dictionary optimization can be said to be an optimization problem of obtaining the first dictionary conversion and the second dictionary conversion that minimize the dictionary optimization loss.

[0104] The dictionary optimization model obtains a word information vector after conversion by the first dictionary conversion obtained by dictionary optimization. The word information vector after conversion shows a word vector with higher sparsity than before conversion.

[0105] <Regarding First Dictionary Loss and Affine Transformation> The first dictionary loss and affine transformation will be described. In dictionary optimization, optimization is performed to reduce the dictionary optimization loss, so optimization is performed to reduce the first dictionary loss. Reducing the first dictionary loss means reducing the difference between the dictionary matrix before the affine transformation and the dictionary matrix after the affine transformation.

[0106] The reason for performing the affine transformation will be explained below. As described above, the dictionary optimization model is a mathematical model that updates word information vectors to increase the sparsity of word vectors. The dictionary matrix is ​​a matrix in which word vectors are arranged in a predetermined direction, either row-wise or column-wise.

[0107] Therefore, if an affine transformation is performed on this, the orientation of the vectors associated with words will be changed. Changing the orientation of vectors can increase sparsity. For example, an affine transformation can cause two vectors that were pointing in roughly the same direction to become orthogonal. For this reason, an affine transformation is performed in dictionary optimization.

[0108] The reason for reducing the first dictionary loss is that the greater the difference between before and after the affine transformation, the greater the possibility that the relationship between the similarity of word meanings and the small difference in word vectors will be lost.

[0109] <Regarding Second Dictionary Loss and Linear Transformation> The second dictionary loss and linear transformation will be described. In dictionary optimization, optimization is performed to reduce the dictionary optimization loss, so optimization is performed to reduce the second dictionary loss. Reducing the second dictionary loss means reducing the difference between the first transformed dictionary matrix and the second transformed dictionary matrix, and as described above, the second transformed dictionary matrix is ​​the result of performing a linear transformation on the first transformed dictionary matrix.

[0110] Simultaneous differential equations, such as mathematical expressions that express coupled vibrations, are a set of multiple differential equations expressed by linear combinations. Such simultaneous differential equations can be expressed as vector differential equations by using a coefficient matrix. A matrix such as a coefficient matrix is ​​a linear transformation of the vector on which it acts (for simplicity, the following explanation will be given assuming that the coefficient matrix acts from the left).

[0111] As is clear from these mathematical phenomena, a linear transformation is a process that generates one linear combination for each row of a coefficient matrix when the destination of the operation is a vector. Since a matrix is ​​a set of vectors, if the destination of a linear transformation is a matrix, as is clear from the rules of matrix operations, a linear combination is generated for each pair of the destination matrix column and the matrix row representing the linear transformation.

[0112] Therefore, since the first transformed dictionary matrix is ​​a set of word vectors, it can be said that the second transformed dictionary matrix is ​​a matrix that represents a linear combination that approximates the first transformed dictionary matrix.

[0113] Therefore, reducing the second dictionary loss means increasing the degree of approximation when each word vector is expressed as a linear combination of other word vectors. The higher the degree of approximation of the linear combination, the higher the accuracy of estimation by, for example, the above-mentioned candidate estimation model.

[0114] <About Element Loss> The smaller the element loss, the fewer word vectors there are when each word vector is expressed as a linear combination of other word vectors. The reason why it is preferable to use fewer word vectors in the linear combination is the same as the reason why it is desirable to reduce word loss. That is, the user's effort in selecting words to input to the result generation model is reduced if the candidate prediction model predicts fewer candidates rather than predicting many candidates.

[0115] Note that the optimization for reducing the loss in the second dictionary and the optimization for reducing the loss in the number of elements do not necessarily have to be used only to improve the estimation accuracy of the candidate estimation model. The optimization for reducing the loss in the second dictionary and the optimization for reducing the loss in the number of elements can also be used to improve the accuracy of the technology described in Reference 4 below, for example.

[0116] Reference 4: Mairal, Julien, et al. "Online dictionary learning for sparse coding." Proceedings of the 26th annual international conference on machine learning. 2009.

[0117] The first dictionary constraint is a constraint that when each word vector is expressed as a linear combination, it must be expressed only in terms of other word vectors. If each word vector is expressed in terms of all word vectors, including itself, the linear combination with the highest degree of approximation will naturally be a linear combination expressed only in terms of itself, the vector to be approximated. Therefore, the first dictionary constraint exists to prevent this phenomenon from occurring.

[0118] The reason why sparsity increases will now be explained. By optimizing to reduce the first dictionary loss, the matrix D is not fixed but changes, as shown by affine transformations, etc. Therefore, optimization to reduce the first dictionary loss allows optimization of a wider searchable space of vectors W than dictionary optimization in which the matrix D is fixed. As a result, a word vector space with even higher sparsity is obtained. This is the reason why sparsity increases.

[0119] <Example of Dictionary Optimization Loss Using a Formula> The dictionary optimization loss, which is a loss obtained by adding a first dictionary constraint to a loss including the first dictionary loss, the second dictionary loss, and the number of elements loss, may be, for example, a loss obtained by adding a first dictionary constraint to a linear sum of the first approximation loss, the second dictionary loss, and the number of elements loss. Hereinafter, the loss obtained by adding a first dictionary constraint to a linear sum of the first approximation loss, the second dictionary loss, and the number of elements loss is referred to as the first dictionary optimization loss. The first dictionary optimization loss is expressed, for example, by the following formula (10).

[0120]

[0121] D ｃｌｉｐ represents the dictionary matrix. D represents the first transformation dictionary matrix. W is a matrix representing the second dictionary transformation. Therefore, the first term in equation (10) represents the second dictionary loss, and the third term represents the first dictionary loss. The second term in equation (10) represents the element count loss. wｉ，ｊ represents the element in row i and column j of matrix W. μ represents the balance parameter of the third term. F represents the Frobenius norm.

[0122] The dictionary optimization loss may be, for example, a loss obtained by adding a condition that the elements of the matrix W are non-negative to the first dictionary optimization loss (hereinafter referred to as a "second dictionary optimization loss"). The dictionary optimization problem using the second dictionary optimization loss is solved under the condition that the elements of the matrix W are non-negative. The second dictionary optimization loss is expressed, for example, by the following equation (11).

[0123]

[0124] The effect of the weights in the matrix W being non-negative is similar to the effect of the condition in the second word loss that the weights w are positive.

[0125] The dictionary optimization loss may be, for example, the second dictionary optimization loss plus one of the predetermined conditions that the sum of the elements in each row of matrix W is 1 or the sum of the elements in each column of matrix W is 1 (hereinafter referred to as the "third dictionary optimization loss").

[0126] The dictionary optimization problem using the third dictionary optimization loss is solved under the condition that the weights of the matrix W are non-negative and also under one of the following predetermined conditions: the sum of the elements of each row of the matrix W is 1, and the sum of the elements of each column of the matrix W is 1. The third dictionary optimization loss is expressed, for example, by the following equation (12).

[0127]

[0128] In the example of equation (12), the sum of the elements in each row of the matrix is ​​1. The effect of the condition that the sum of the elements in each row of matrix W is 1 and the condition that the sum of the elements in each column of matrix W is 1 is similar to the effect of the condition that the sum of the weights in the first linear combination is 1 in the third word loss.

[0129] <Example of optimization> In dictionary optimization, the dictionary optimization model may use any optimization method to obtain the first dictionary conversion and the second dictionary conversion that minimize the value of the dictionary optimization loss based on word vector information.

[0130] For example, any of equations (10) to (12) can be expressed as the following equation (13).

[0131]

[0132] Here, C represents the following formula (14) when formula (13) represents formula (10), represents the following formula (15) when formula (13) represents formula (11), and represents the following formula (16) when formula (13) represents formula (12).

[0133]

[0134]

[0135]

[0136] For example, Equation (13) may be optimized by alternating between changing only one of variables W and D. In such a case, variable W may be optimized by, for example, the alternating direction multiplier method described in Reference 2, and variable D may be optimized by the least squares method. Such an optimization algorithm may be, for example, the algorithm shown in Equation (17) below.

[0137]

[0138] In equation (17), proj represents the distance projection onto the convex set C. Note that when the dictionary optimization loss is the first dictionary optimization loss, the distance projection is x ｉ，ｉ It may be obtained by setting x = 0. ｉ，ｉ represents the diagonal elements of the weight matrix.

[0139] If the dictionary optimization loss is the second dictionary optimization loss, the distance projection is x ｉ，ｉ = 0, and then other x ｉ，ｊ may be obtained by clipping x to a non-negative value.ｉ，ｊ represents the off-diagonal elements of the weight matrix.

[0140] If the dictionary optimization loss is the second dictionary optimization loss, the distance projection is x ｉ，ｉ = 0 and then projecting each column onto a probability simplex (see Reference 3). In equation (17), the parameter γ is a predetermined positive value and represents the step size.

[0141] 4 is a diagram showing an example of the flow of processing executed by the optimization device 1 in the modified example. The control unit 11 executes a dictionary optimization model (step S201). By executing the dictionary optimization model, first dictionary conversion and second dictionary conversion that minimize the dictionary optimization loss value are obtained, and a word information vector after conversion using the obtained first dictionary conversion is obtained.

[0142] In this way, the optimization device 1 includes a control unit 11 that executes the dictionary optimization model. By executing the dictionary optimization model, a word information vector after conversion by the first dictionary conversion is obtained, so the optimization device 1 can reduce the effort required to use a mathematical model that outputs a result according to an input word.

[0143] The optimization device 1 may be implemented using a plurality of information processing devices connected to each other via a network so that they can communicate with each other. In this case, the processes executed by the control unit 11 may be distributed among the plurality of information processing devices.

[0144] Note that all or part of the functions of the control unit 11, the interface unit 12, and the storage unit 13 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The program may be transmitted via a telecommunications line.

[0145] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0146] REFERENCE SIGNS LIST 1... Optimization device, 11... Control unit, 12... Interface unit, 13... Storage unit, 91... Processor, 92... Memory

Claims

1. An optimization device comprising: a control unit that executes a candidate estimation model, which is a mathematical model that estimates word candidates to be input to a result generation model, which is a mathematical model that outputs a result according to an input word, according to a target word, which is a given word; wherein the candidate estimation model is a mathematical model that solves an optimization problem to obtain weights of the linear combination that minimizes a loss value including: a quantity that represents a difference between a linear combination of a target vector, which is a vector representing the target word, and a non-target vector, which is a vector representing another word different from the target word, the smaller the quantity, the smaller the quantity, and a quantity that represents the number of non-target vectors whose weight in the linear combination is not zero, the smaller the quantity, based on word vector information that indicates a predetermined one-to-one correspondence between one or more given words and vectors in a predetermined vector space, and the target word.

2. The optimization device according to claim 1, wherein the optimization problem is solved under the condition that weights in the linear combination are non-negative.

3. An optimization method comprising: a control step of executing a candidate estimation model, which is a mathematical model that estimates word candidates to be input to a result generation model, which is a mathematical model that outputs a result according to an input word, according to a target word that is a given word, wherein the candidate estimation model is a mathematical model that solves an optimization problem to obtain each weight of the linear combination that minimizes a loss value including: a quantity that represents a difference between a linear combination of a target vector that is a vector representing the target word and a non-target vector that is a vector representing another word different from the target word, the smaller the quantity, and a quantity that represents the number of non-target vectors whose weight in the linear combination is not zero, the smaller the quantity, based on word vector information that indicates a predetermined one-to-one correspondence between one or more given words and vectors in a predetermined vector space, and the target word.

4. A program for causing a computer to function as the optimization device according to claim 1.

Images