Type vector generation device, type vector generation method, and program

Abstract

In the present invention, for each type xi (1≤i≤p) for which a token differs in an inputted token string that has been, a fundamental vector si of a q number of dimensions comprising an element corresponding to each type and an element corresponding to an attribute, a p×q matrix S=(si) is generated, a p×r matrix U=ST is aquired from a q×r matrix T in which embedded vectors tj, for which each element value of an r number of dimensions is a parameter variable, are arranged as a 1×r matrix corresponding to a Jth element of a fundamental vector, a p×r matrix W=SV is acquired from a q×r matrix V in which embedded vectors vj, for which each element value of the r number of dimensions is a parameter variable, are arranged as a 1×r matrix corresponding to the Jth element of the fundamental vector, a type corresponding to any token y in the token string is defined as xn, a string of a type corresponding to a token near y in the token string is defined as xm_g, a probability p(xn|xm_g) is acquired, and each element value of the matrices T, V for which the product of the probability for any y in the token string is maximized is obtained.

Description

[Technical Field]

[0001] The present invention relates to a type vector generation device, a type vector generation method, and a program. [Background technology]

[0002] The units of words or characters obtained by dividing a string of characters, such as a natural language, are called tokens.

[0003] Tokens can be defined as a set of a string and a set of attributes, as shown in Figure 1. In this case, the difference between tokens refers to a group of tokens whose string and attribute set are identical, and this is called a type. In Figure 1, tokens 3 and 5 have the same string and attribute set, so their corresponding types are the same. Tokens 1 and 3 have the same string, but their attribute sets are different, so their corresponding types are different.

[0004] Sometimes tokens are defined as strings only. In this case, token differences refer to groups of tokens with identical strings, and these are called types. In Figure 1, tokens 1, 3, and 5 have identical strings, so their corresponding types are the same.

[0005] Regarding token types, which represent differences between tokens, one method for generating type vectors to quantify the semantic similarity between types is the word2vec-based method described in Non-Patent Document 1.

[0006] Furthermore, there are methods that generate language models using feedforward neural language models and recurrent neural language models described in Non-Patent Document 2, Transformers described in Non-Patent Document 3, and BERT described in Non-Patent Document 4, thereby determining the parameter values ​​of some of the embedding matrices and defining each type of vector.

[0007] These conventional methods only use strings for tokens and types. Then, as shown in Figure 2, each element corresponds to a type, and a vector is considered as a 1×p matrix with dimensions equal to the number of types (let's call it p), and each type x i For (1≦i≦p), the x of the vector i Associate a 1-hot vector with all elements set to 1 and all other elements set to 0. Multiply this 1-hot vector from the right by an embedding matrix, where each element of the p×r matrix is ​​a parameter variable, to obtain an r-dimensional vector, which is then assigned to x i Let this be the type vector.

[0008] When we consider a range of consecutive token sequences, or a range such as a single sentence, the tokens (or the types corresponding to those tokens) within that range are said to co-occur, or to be in a co-occurring relationship.

[0009] Conventional methods determine the type vector by finding the parameter value that maximizes the probability of predicting tokens that have a co-occurrence relationship. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, "Efficient estimation of word representations in vector space", ICLR, 2013 [Non-Patent Document 2] Yuta Tsuboi, Yuya Unno, Jun Suzuki, "Natural Language Processing with Deep Learning," Kodansha, 2017. [Non-Patent Document 3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, L'ukasz Kaiser (L' is L / (Kreska)), Illia Polosukhin, "Attention is All you Need", Advances in Neural Information Processing Systems 30, pp.5998 - 6008, 2017 [Non - Patent Document 4]<​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​ However, z1,z m If they do not co-occur and cannot be linked by a chain of co-occurring types as listed above, then z1,z m Although they are semantically similar, z1, z m There was a problem in that the vectors were not always close to each other.

[0015] This invention has been made in view of the above points, and aims to bring together vectors of each type that are semantically similar but do not co-occur and cannot be linked by a chain of co-occurring types. [Means for solving the problem]

[0016] Therefore, to solve the above problem, the type vector generator generates a type x which is the difference between each token in the input token sequence. i For (1≦i≦p), the basis vector s of dimension q is a 1×q matrix consisting of elements corresponding to each type and elements corresponding to attributes. i =(s ij ) 1≦j≦q Generate a p×q matrix S=(s i ) 1≦i≦p A type of base vector generation means configured to generate a base vector, and an embedded vector t whose element values ​​are parameter variables, where each element value is a 1×r matrix of dimension r, corresponding to the Jth element of the base vector. j =( t jk ) 1≦k≦r The input matrix T=(t) is formed by arranging the following elements: j ) 1≦j≦q From this, the input p×r matrix U=ST=(u i ) 1≦i≦p =( u ik ) 1≦i≦p,1≦k≦r The embedded vector v is a 1×r matrix of dimension r, where each element value is a parameter variable, corresponding to the Jth element of the aforementioned basis vector. j =(v jk ) 1≦k≦r The output q×r matrix V=(v j ) 1≦j≦qTherefore, the output p×r matrix W=SV=(w i ) 1≦i≦p =(w ik ) 1≦i≦p,1≦k≦r A parameter matrix acquisition means configured to take the following: For any token y in the token sequence, the type corresponding to y is x n Let x be a sequence of the type corresponding to the tokens in the vicinity of y in the sequence of tokens. m_g Let (1 ≤ g ≤ h), and the probability p(x n │x m_g The system includes a first optimal parameter calculation means configured to take (1 ≤ g ≤ h) and find the element values ​​of matrices T and V that maximize the product of the probabilities for any token y in the token sequence. [Effects of the Invention]

[0017] It is possible to bring together vectors of each type that are semantically similar but do not co-occur and cannot be linked by a chain of co-occurring types. [Brief explanation of the drawing]

[0018] [Figure 1] This figure shows an example of a token consisting of a string and a set of attributes. [Figure 2] This diagram illustrates how the number of dimensions corresponds to the number of types of a vector. [Figure 3] This figure shows an example of the hardware configuration of the type vector generation device 10 in the first embodiment. [Figure 4] This figure shows an example of the functional configuration of the type vector generation device 10 in the first embodiment. [Figure 5] This diagram shows the relationships between matrices S, T, U, V, and W. [Figure 6] This is a diagram showing an example of a thesaurus. [Figure 7] This figure shows an example of si in a thesaurus where only the element values ​​of the node to which xi belongs are set to 1. [Figure 8]This figure shows an example of si in a thesaurus where the values ​​of the elements from the node to which xi belongs to the root node are all set to 1. [Figure 9] This figure shows the first example of each type being considered an element corresponding to an attribute in SI. [Figure 10] This figure shows a second example where each type is considered an element corresponding to an attribute in SI. [Figure 11] This figure shows a third example where each type is considered an element corresponding to an attribute in SI. [Figure 12] This is a diagram illustrating the process performed by the first optimal parameter calculation means 13. [Figure 13] This figure shows an example of the functional configuration of the type vector generation device 10 in the second embodiment. [Figure 14] This is a diagram illustrating the process performed by the second optimal parameter calculation means 14. [Figure 15] This figure shows an example of the functional configuration of the type vector generation device 10 in the third embodiment. [Figure 16] This is a diagram illustrating the processing performed by the neural network processing means 15. [Modes for carrying out the invention]

[0019] Embodiments of the present invention will be described below with reference to the drawings.

[0020] In this embodiment, the units of words or characters obtained by dividing a string of characters, such as natural language, are called tokens.

[0021] Tokens can be defined as a set of a string and a set of attributes, as shown in Figure 1. In this case, the difference between tokens refers to a group of tokens whose string and attribute set are identical, and this is called a type. In Figure 1, tokens 3 and 5 have the same string and attribute set, so their corresponding types are the same. Tokens 1 and 3 have the same string, but their attribute sets are different, so their corresponding types are different.

[0022] Sometimes tokens consist only of strings. In this case, token differences refer to groups of tokens with identical strings, and these are called types. In Figure 1, tokens 1, 3, and 5 have identical strings, so their corresponding types are the same.

[0023] [First Embodiment of the Invention] Figure 3 shows an example of the hardware configuration of the type vector generator 10 in the first embodiment. The type vector generator 10 in Figure 3 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, etc., which are all interconnected by bus B.

[0024] The program that implements the processing in the type vector generator 10 is provided on a recording medium 101 such as a CD-ROM. When the recording medium 101 containing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101; it may also be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files and data.

[0025] The memory device 103 reads and stores a program from the auxiliary storage device 102 when a program startup instruction is received. The processor 104 is either a CPU or a GPU (Graphics Processing Unit), or both a CPU and a GPU, and executes the functions related to the type vector generator 10 according to the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0026] Figure 4 shows an example of the functional configuration of the type vector generation device 10 in the first embodiment. In Figure 4, the type vector generation device 10 includes a type base vector generation means 11, a parameter matrix acquisition means 12, and a first optimal parameter calculation means 13. Each of these means is realized by a process in which one or more programs installed in the type vector generation device 10 are executed by the processor 104.

[0027] The processing details of the type-based vector generation means 11 in the configuration diagram of Figure 4 will be described below.

[0028] The type-based vector generation means 11 takes a token sequence as input and generates a type x which is the difference between the tokens in the token sequence. i For (1≦i≦p), the basis vector s of dimension q is a 1×q matrix consisting of elements corresponding to each type and elements corresponding to attributes. i =(s ij ) 1≦j≦q Generate a p×q matrix S=(s i ) 1≦i≦p Generates.

[0029] Details are provided below.

[0030] Each type x i (1≦i≦p) can be a pair of a string and a set of attributes, or it can be a string only.

[0031] Figure 5 illustrates matrix S. The i-th row of matrix S in Figure 5 is s i That is the case.

[0032] s i It can have elements of any type. The types of the elements of a vector are each type x i Independent of the configuration information (1≦i≦p), it may be a set of a string and an attribute group, or a string only. In the matrix S in Figure 5, each type x i Since both (1≦i≦p) and the types of elements in the vector are either a set of a string and an attribute group, or a string only, the number of elements corresponding to each type is p, which is the same as the number of rows.

[0033] x i In contrast, s i The value of the corresponding type of element in is set to 1, and the value of the other type of element is set to 0. Typically, values ​​are set in this way, but any real number is also acceptable. Also, in Figure 5, x i In contrast, s i The element of the corresponding type in the expression is designated as the i-th element, but the elements do not necessarily have to be arranged in this order.

[0034] s i x has elements corresponding to its attributes. i Among the attributes held by, there may be some that are not elements, and also x i It may have attributes other than those it possesses as an element.

[0035] As attributes to be used as elements, each node in the thesaurus shown in Figure 6 may be taken. In Figure 7, all nodes are used as elements, x i The element values ​​of the node to which x belongs are set to 1, and the element values ​​of all other nodes are set to 0. In Figure 8, x i The element values ​​of nodes on the path from the node to which the element belongs to the root node are set to 1, and the element values ​​of all other nodes are set to 0.

[0036] Each attribute can be of a specific type. i Independent of the configuration information (1≦i≦p), it may be a set of a string and a group of attributes, or it may be a string only.

[0037] As attributes to be used as elements, we take each type as shown in Figure 9, x i Alternatively, we could set the value of elements of a higher-level concept to 1, and the value of elements of all other types to 0.

[0038] As attributes to be used as elements, we take each type as shown in Figure 10, x iAlternatively, elements of a sub-concept could be assigned a value of 1, while elements of other types could be assigned a value of 0.

[0039] As attributes to be used as elements, we take each type as shown in Figure 11, x i Alternatively, we could set the value of elements of the opposite type to 1, and the value of elements of all other types to 0.

[0040] The values ​​of the elements corresponding to attributes are not limited to 0 or 1; they can be any real number.

[0041] Although I have listed various types of elements and attributes above, s i It may have multiple types of attributes as elements.

[0042] All x i For (1≦i≦p), the elements corresponding to the attribute whose value is 0 are s i You may also remove it from the set. In this case, if there is a corresponding element, s i The number of dimensions decreases.

[0043] The processing details of the parameter matrix acquisition means 12 in the configuration diagram of Figure 4 will be described below.

[0044] The parameter matrix acquisition means 12 obtains an embedded vector t, which is a 1×r matrix for input corresponding to the Jth element of the basis vector, where each element value of dimension r is a parameter variable. j =( t jk ) 1≦k≦r The input matrix T=(t) is formed by arranging the following elements: j ) 1≦j≦q From this, the input p×r matrix U=ST=(u i ) 1≦i≦p =( u ik ) 1≦i≦p,1≦k≦r The embedded vector v is a 1xr matrix for output, where each element value of dimension r corresponds to the Jth element of the basis vector, and the parameter variables are the values ​​of the elements of the resulting matrix. j =(v jk ) 1≦k≦r The output q×r matrix V=(v j )1≦j≦q From this, for the output p×r matrix W = SV=(w i ) 1≦i≦p =(w ik ) 1≦i≦p,1≦k≦r is taken. Herein, "take" or "taken" means that the corresponding matrix can be calculated or conceptually considered.

[0045] Matrices T, U, V, and W are illustrated in FIG. 5.

[0046] [[ID=1​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Take c tokens before and after y respectively b-c , y b-c+1 , …, y b-2 , y b-1 , y b+1 , y b+2 , …, y b+c-1 , y b+c may be used as the neighborhood of y b .

[0053] Let the sequence of types corresponding to the tokens in the neighborhood of y be x m_g When (1 ≤ g ≤ h), there may be multiple identical types in this type sequence

[0054] As shown in FIG. 12, the first optimal parameter calculation means 13 calculates the sum of the vectors of the types in the neighborhood

[0055]

Number

[0056] The first optimal parameter calculation means 13 obtains the values of each element of the matrices T and V for which the product of the above probabilities derived from any token y in the token sequence (i.e., the product of the probabilities derived for each token) is maximized. This is equivalent to obtaining the values of each element of the matrices T and V for which the sum of the following negative log-likelihoods derived from any token y in the token sequence (i.e., the sum of the negative log-likelihoods derived for each token) is minimized

[0057]

Number

[0058] [Second Embodiment of the Invention] Next, a second embodiment will be described. The differences between the second embodiment and the first embodiment will be described. Points in the second embodiment that are not specifically mentioned may be the same as in the first embodiment.

[0059] Figure 13 shows an example of the functional configuration of the type vector generation device 10 in the second embodiment. In Figure 13, the same means as in Figure 4 are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0060] A second embodiment of the present invention takes a token sequence as input and includes a type-based vector generation means 11 and a parameter matrix acquisition means 12 of the first embodiment of the present invention, and performs their respective processes. Furthermore, the type vector generation device 10 of the second embodiment has a second optimal parameter calculation means 14 instead of a first optimal parameter calculation means 13. The second optimal parameter calculation means 14 is realized by a process in which one or more programs installed in the type vector generation device 10 are executed by a processor 104.

[0061] The processing details of the second optimal parameter calculation means 14 in the configuration diagram of Figure 13 will be described below.

[0062] The second optimal parameter calculation means 14 determines the type corresponding to any token y in the token sequence as x n Let x be a sequence of the type corresponding to the tokens in the vicinity of y in the sequence of tokens. m_g Let (1 ≤ g ≤ h), and the probability

[0063]

number

[0064] Details are provided below.

[0065] The token sequence is y1, y2, ..., y a When this is the case, any token y in the token sequence b Regarding y b y obtained by taking c tokens before and after b-c ,y b-c+1 ,…,y b-2 ,y b-1 ,y b+1 ,y b+2 ,…,y b+c-1 ,y b+c to y b It may also be considered a neighborhood.

[0066] A sequence of type corresponding to tokens in the neighborhood of y is x m_g When (1 ≤ g ≤ h), this type sequence can contain multiple identical types.

[0067] As shown in Figure 14, the second optimal parameter calculation means 14 is type x n The vector corresponding to t w n Then, the inner product of each row of matrix U is calculated, and the inner product is probabilistically calculated using the softmax function. Type x m_g The vector corresponding to u m_g The value obtained by multiplying the probability of the inner product with by all g(1≦g≦h) is of type x n This represents the probability of predicting the type sequence in the neighborhood.

[0068] The second optimal parameter calculation means 14 finds the element values ​​of matrices T and V that maximize the product of the above probabilities derived from any token y in the token sequence. This is equivalent to finding the element values ​​of matrices T and V that minimize the sum of the following negative log-likelihoods derived from any token y in the token sequence.

[0069]

number

[0070] [Third Embodiment of the Invention] Next, a third embodiment will be described. The differences between the third embodiment and the first embodiment will be described. Points not specifically mentioned in the third embodiment may be the same as in the first embodiment.

[0071] Figure 15 shows an example of the functional configuration of the type vector generation device 10 in the third embodiment. In Figure 15, the same means as in Figure 4 are denoted by the same reference numerals, and their descriptions are omitted as appropriate.

[0072] A third embodiment of the present invention takes a token sequence as input and includes the type-based vector generation means 11 and parameter matrix acquisition means 12 of the first embodiment of the present invention, and performs their respective processes. Furthermore, the type vector generation device 10 of the second embodiment has a neural network processing means 15 instead of the first optimal parameter calculation means 13. The neural network processing means 15 is realized by having one or more programs installed in the type vector generation device 10 execute processes on the processor 104.

[0073] The processing details of the neural network processing means 15 in the configuration diagram of Figure 15 will be described below.

[0074] The neural network processing means 15, for the input token sequence E, assigns each token y in E to the type x corresponding to y. m The corresponding vector u m Convert to (type x corresponding to y from matrix U) m The corresponding vector u m (obtain the vector u) and process it in a neural network to derive a vector of dimension r. m(By inputting the sequence into a neural network, a vector of dimension r is obtained from the neural network), let f be an arbitrary vector of dimension r derived from this, and let the vector d=(d i ) 1≦i≦p Take (prepare as a parameter vector), and for each type x of f n The probability of outputting

[0075]

number

[0076] Details are provided below.

[0077] Examples of neural network processing include the feedforward neural language model and recurrent neural language model described in Non-Patent Document 2, the Transformer described in Non-Patent Document 3, and BERT described in Non-Patent Document 4, where the input tokens are embedded into an embedding vector (of type x as described above). m The corresponding vector u m After being converted to ), processing continues until just before the output layer. As described above, the input to this process is an embedding vector, and the output to this process is a vector of dimension r.

[0078] For neural networks that perform sequential processing, such as recursive neural language models, the input token sequence E may consist of only one token.

[0079] As shown in Figure 16, the neural network processing means 15 transposes the matrix W. t For each column of W, the inner product of that column and vector f is calculated, the values ​​of the corresponding elements of vector d are added together, and the resulting value is probabilistically calculated using the softmax function.

[0080] In the third embodiment of the present invention, each element value of matrices T and V is a parameter variable, and there are other parameter variables in the neural network's processing. These parameter variables are learned as follows.

[0081] The neural network processing means 15 assigns the type of the correct answer to each f as x n When this is the case, type x n We find the parameter variable that maximizes the value obtained by multiplying the output probability (the probabilistic value explained using Figure 16) by all f, and then multiplying this value by all token sequences E (we find the values ​​of each element of matrices T and V, and the values ​​of the other parameter variables, so that this value is maximized).

[0082] [Fourth Embodiment of the Present Invention] In any of the first to third embodiments of the present invention described above, the row vector u of the matrix U obtained by the parameter matrix acquisition means 12 i and the row vector w of matrix W i This corresponds to the basis vector s i As the number of non-zero elements increases, the embedded vector t corresponding to each non-zero element is added. j Ya v j As the number of elements increases, the vector length tends to become larger.

[0083] As the vector lengths increase, the distance between vectors may increase for pairs of vectors that maximize their dot product, potentially leading to an inappropriate arrangement.

[0084] To correct this, the row vector u of matrix U i and the row vector w of matrix W i These correspond to the respective basis vectors s i Embedding vector t corresponding to the non-zero elements j The center of gravity of v j It should be the center of gravity.

[0085] Based on the above, in the fourth embodiment of the present invention, the parameter matrix acquisition means 12 performs the following processing in any of the first to third embodiments of the present invention.

[0086] Let U be a matrix U = ST = (u i ) 1≦i≦p =( u ik ) 1≦i≦p,1≦k≦r After taking it as such, each element u of matrix U ik against

[0087]

number

[0088]

number

[0089] In any of the first to fourth embodiments of the present invention described above, after determining the element values ​​of matrices T and V, the element values ​​of matrices U and W are determined (U and W are determined from S and T and V). At this time, each row u of matrices U and W i ,w i However, the corresponding type x i This becomes a vector with defined element values.

[0090] In any of the first to fourth embodiments of the present invention described above, the parameter variables (t) of each element of the matrix T in the parameter matrix acquisition means 12 jk ) and the corresponding identical element parameter variable (v) of matrix V jk ) is identical to, that is, matrix V = (v jk ) 1≦j≦q,1≦k≦r The matrix T=(t jk ) 1≦j≦q,1≦k≦rIt may also be constructed such that (i.e., V=T).

[0091] In the embodiments of the present invention described above, each vector and each matrix may be transposed.

[0092] As described above, each of the above embodiments is a type x i The basis vectors corresponding to (1≦i≦p) are characterized by the fact that they retain not only the elements corresponding to each type, as held in conventional methods, but also elements corresponding to attributes.

[0093] When we consider a range of consecutive token sequences, or a range such as a single sentence, the tokens (or the types corresponding to those tokens) within that range are said to co-occur, or to be in a co-occurrence relationship.

[0094] In each of the above embodiments, the vectors of the type that are in a co-occurrence relationship (u i lol i ) tends to be close because it seeks parameter values ​​that maximize the dot product.

[0095] Two semantically similar types z1, z m Regarding z1,z m If they co-occur, z1, z m The vectors will become closer.

[0096] z1,z m Even if z1 and z2 do not co-occur, z1 and z2 co-occur, z2 and z3 co-occur, ... z m-1 ,z m If they co-occur, then the vectors of the co-occurring types tend to be close together, so z1, z m The vectors will become closer.

[0097] Two semantically similar types z1, z m Regarding z1,z m Let's consider the case where the elements do not co-occur and cannot be linked by the types of co-occurring elements listed above.

[0098] In each of the above embodiments, the attributes of the non-zero elements of the base vector of the type corresponding to the token within the range co-occur with each other. Then, the embedding vector (t) corresponding to the co-occurring attributes j Ya v j ) is the vector (u i lol i The parameter values ​​tend to be close because they are sought to maximize the dot product of ).

[0099] In the following description, "attribute" refers to the non-zero element attribute of the type's base vector.

[0100] z1,z m However, if they have the same attribute, z1, z m The vector will be close because it contains the embedding vector of the attribute in question.

[0101] z1,z m If they do not have the same attribute, attribute a1 of z1 co-occurs with attribute a2, a2, a3 co-occurs, ..., a m-1 ,a m co-occurs (where the types with the same attribute may differ depending on the co-occurrence range. For example, the types with a2 may differ in the co-occurrence range of a1, a2 and the co-occurrence range of a2, a3), z m is attribute a m If a certain property is present, the embedding vectors of co-occurring attributes tend to be close together, and therefore z1, z contain the embedding vector of that attribute. m The vectors will become closer.

[0102] In this way, two semantically similar types z1, z m Regarding this, even in cases where the vectors do not become close using conventional methods, the above embodiments achieve the effect of making the vectors closer by considering the co-occurrence of attributes. In order to obtain such an effect, it is necessary that semantically similar types of attributes are either identical or likely to co-occur directly or indirectly.

[0103] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims. [Explanation of Symbols]

[0104] 10-Type Vector Generator 11 Type-based vector generation means 12. Means for obtaining the parameter matrix 13. First optimal parameter calculation means 14. Second optimal parameter calculation means 15 Neural network processing means 100 drive unit 101 Recording media 102 Auxiliary storage device 103 Memory device 104 Processors 105 Interface device B Bus

Claims

1. Each type x represents the difference between tokens in the input token sequence. ｉ For (1 ≤ i ≤ p), the basis vector s of dimension q is a 1 × q matrix consisting of elements corresponding to each type and elements corresponding to attributes. ｉ = (s ｉｊ ) １≦ｊ≦ｑ Generate a p × q matrix S = (s ｉ ) １≦ｉ≦ｐ A type-based vector generation means configured to generate, An embedding vector t, which is a 1×r matrix for input corresponding to the J-th element of the basis vector and each element value of dimension r is a parameter variable ｊ = (t ｊｋ ), １≦ｋ≦ｒ Take the q×r matrix T for input arranged as (t ｊ ), １≦ｊ≦ｑ and from it, take the p×r matrix U = ST = (u ｉ ), １≦ｉ≦ｐ = (u ｉｋ ), １≦ｉ≦ｐ，１≦ｋ≦ｒ and an embedding vector v, which is a 1×r matrix for output corresponding to the J-th element of the basis vector and each element value of dimension r is a parameter variable ｊ = (v ｊｋ ), １≦ｋ≦ｒ Take the q×r matrix V for output arranged as (v ｊ ), １≦ｊ≦ｑ and from it, take the p×r matrix W = SV = (w ｉ ), １≦ｉ≦ｐ = (w ｉｋ ), １≦ｉ≦ｐ，１≦ｋ≦ｒ Parameter matrix acquisition means configured to obtain For any token y in the token sequence, the type corresponding to y is x ｎ Let x be a sequence of the type corresponding to a token in the vicinity of y in the sequence of tokens. ｍ＿ｇ Let (1 ≤ g ≤ h), and the probability [Math 1] A first optimal parameter calculation means is configured to take a value and find the element values ​​of matrices T and V that maximize the product of the probabilities for any token y in the token sequence, A type vector generation device characterized by comprising the following features.

2. Each type x represents the difference between tokens in the input token sequence. ｉ For (1 ≤ i ≤ p), the basis vector s of dimension q is a 1 × q matrix consisting of elements corresponding to each type and elements corresponding to attributes. ｉ = (s ｉｊ ) １≦ｊ≦ｑ Generate a p × q matrix S = (s ｉ ) １≦ｉ≦ｐ A type-based vector generation means configured to generate, The embedded vector t is a 1xr matrix with dimension r, corresponding to the Jth element of the aforementioned foundation vector, where each element value is a parameter variable. ｊ = (t ｊｋ ) １≦ｋ≦ｒ The input matrix T = (t ｊ ) １≦ｊ≦ｑ From this, the input p × r matrix U = ST = (u ｉ ) １≦ｉ≦ｐ = (u ｉｋ ) １≦ｉ≦ｐ，１≦ｋ≦ｒ The embedded vector v is a 1xr matrix of dimension r, where each element value is a parameter variable, corresponding to the Jth element of the aforementioned basis vector. ｊ = (v ｊｋ ) １≦ｋ≦ｒ The output matrix V = (v ｊ ) １≦ｊ≦ｑ Therefore, the p × r matrix for output W = SV = (w ｉ ) １≦ｉ≦ｐ = (w ｉｋ ) １≦ｉ≦ｐ，１≦ｋ≦ｒ A parameter matrix acquisition means configured to take the following, For any token y in the token sequence, the type corresponding to y is x ｎ Let x be a sequence of the type corresponding to a token in the vicinity of y in the sequence of tokens. ｍ＿ｇ Let (1 ≤ g ≤ h), and the probability [Math 2] A second optimal parameter calculation means is configured to take a value and find the element values ​​of matrices T and V that maximize the product of the probabilities for any token y in the token sequence, A type vector generation device characterized by comprising the following features.

3. Each type x represents the difference between tokens in the input token sequence. ｉ For (1 ≤ i ≤ p), the basis vector s of dimension q is a 1 × q matrix consisting of elements corresponding to each type and elements corresponding to attributes. ｉ = (s ｉｊ ) １≦ｊ≦ｑ Generate a p × q matrix S = (s ｉ ) １≦ｉ≦ｐ A type-based vector generation means configured to generate, The embedded vector t is a 1xr matrix with dimension r, corresponding to the Jth element of the aforementioned foundation vector, where each element value is a parameter variable. ｊ = (t ｊｋ ) １≦ｋ≦ｒ The input matrix T = (t ｊ ) １≦ｊ≦ｑ From this, the input p × r matrix U = ST = (u ｉ ) １≦ｉ≦ｐ = (u ｉｋ ) １≦ｉ≦ｐ，１≦ｋ≦ｒ The embedded vector v is a 1xr matrix of dimension r, where each element value is a parameter variable, corresponding to the Jth element of the aforementioned basis vector. ｊ = (v ｊｋ ) １≦ｋ≦ｒ The output matrix V = (v ｊ ) １≦ｊ≦ｑ Therefore, the p × r matrix for output W = SV = (w ｉ ) １≦ｉ≦ｐ = (w ｉｋ ) １≦ｉ≦ｐ，１≦ｋ≦ｒ A parameter matrix acquisition means configured to take the following, For an input token sequence E, each token y in E corresponds to type x ｍ The corresponding vector u ｍ Convert to a vector, process with a neural network, derive a vector of dimension r, let f be an arbitrary vector of dimension r derived from that vector, and let the element values ​​of dimension p be the parameter variables for the vector d = (d ｉ ) １≦ｉ≦ｐ Take f and for each type x ｎ The probability of outputting [Math 3] A neural network processing means configured to calculate as, A type vector generation device characterized by comprising the following features.

4. The parameter matrix acquisition means determines each element u of matrix U ｉｋ against [Math 4] after updating u ｉｋ Let w be the element of matrix W. ｉｋ against [Math 5] After the update lol ｉｋ It is configured to be such A type vector generation apparatus according to any one of claims 1 to 3, characterized in that it is a type vector generation apparatus.

5. The parameter matrix acquisition means is configured such that the parameter variables of each element of matrix T are the same as the parameter variables of the corresponding elements of matrix V. A type vector generation apparatus according to any one of claims 1 to 3, characterized in that it is a type vector generation apparatus.

6. The parameter matrix acquisition means sets the parameter variables of each element of matrix T to be the same as the parameter variables of the corresponding same element of matrix V, and each element of matrix U and matrix W u ｉｋ against [Math 6] after updating u ｉｋ It is configured to be such A type vector generation apparatus according to any one of claims 1 to 3, characterized in that it is a type vector generation apparatus.

7. For each type x that is a difference of tokens in the input token sequence ｉ For (1 ≤ i ≤ p), a basis vector s of dimension q as a 1 × q matrix consisting of an element corresponding to each type and an element corresponding to an attribute ｉ = (s ｉｊ ) １≦ｊ≦ｑ is generated, and a type basis vector generation procedure for generating a p × q matrix S = (s ｉ ) １≦ｉ≦ｐ is generated An embedding vector t, which is a 1×r matrix for input corresponding to the J-th element of the basis vector and each element value of dimension r is a parameter variable ｊ =(t ｊｋ ), １≦ｋ≦ｒ arrange the q×r matrix T=(t ｊ ) １≦ｊ≦ｑ for input, and from it, take the p×r matrix U = ST=(u ｉ ) １≦ｉ≦ｐ =(u ｉｋ ), １≦ｉ≦ｐ，１≦ｋ≦ｒ and an embedding vector v, which is a 1×r matrix for output corresponding to the J-th element of the basis vector and each element value of dimension r is a parameter variable ｊ =(v ｊｋ ), １≦ｋ≦ｒ arrange the q×r matrix V=(v ｊ ) １≦ｊ≦ｑ for output, and from it, take the p×r matrix W = SV=(w ｉ ) １≦ｉ≦ｐ =(w ｉｋ ), １≦ｉ≦ｐ，１≦ｋ≦ｒ a parameter matrix acquisition procedure, For any token y in the token sequence, the type corresponding to y is x ｎ Let x be a sequence of the type corresponding to a token in the vicinity of y in the sequence of tokens. ｍ＿ｇ Let (1 ≤ g ≤ h), and the probability [Number 7] A first optimal parameter calculation procedure that takes the following and finds the element values ​​of matrices T and V that maximize the product of the probabilities for any token y in the token sequence, A method for generating type vectors, characterized in that a computer performs the following operation.

8. A program characterized by causing a computer to function as a type vector generation device according to any one of claims 1 to 3.

Images