Information processing device, information processing method, and information processing program

Abstract

In the present invention, an acquisition unit acquires a model that uses, as input, first information corresponding to a first object and second information which corresponds to a second object and which is added to the first information, and that outputs third information indicating an inference result with respect to the first information. The model includes an attention mechanism that receives a first input corresponding to the first information and a second input corresponding to the second information. An inference unit: inputs, to the model acquired by the acquisition unit, target first information which is first information serving as an inference target, and target second information which is second information corresponding to the target first information; and causes the model to output target third information which is an inference result with respect to the target first information, thereby generating target third information in which the target second information is reflected by the attention mechanism.

Description

Information processing device, information processing method, and information processing program

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program.

[0002] Conventionally, various machine learning models (also simply referred to as "models"), such as deep learning models, have been proposed. For example, models using attention mechanisms, such as the Transformer model (sometimes simply referred to as "Transformer"), are known (see Non-Patent Documents 1 and 2).

[0003] "Attention Is All You Need." Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. In Proc. NIPS(Neural Information Processing Systems)'17, volume 30, 2017."A Tensorized Transformer for Language Modeling." Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. [online], [searched on May 15, 2024], Internet <URL: https: / / arxiv.org / pdf / 1906.09777>

[0004] However, there is room for improvement in the conventional technology. For example, while the above conventional technology uses an attention mechanism to retrieve information from memory using a query corresponding to an inference target, it may be difficult to properly retrieve information simply by using a query. For example, when performing processing that reflects complex relationships in real-world interactions, the current attention mechanism may not be able to fully reflect those relationships, and in such cases, it may be difficult to perform appropriate inference processing. Therefore, it is desirable to enable inference that properly reflects information other than information input as an inference target using the attention mechanism of a model, for example.

[0005] The present invention has been made in view of the above, and has as its object to perform inference that appropriately reflects information other than information input as an inference target.

[0006] In order to solve the above-mentioned problems and achieve the object, the information processing device of the present invention is characterized by having an acquisition unit that acquires a model that receives first information corresponding to a first object and second information corresponding to a second object and added to the first information as input, and outputs third information that indicates an inference result for the first information, the model including an attention mechanism that accepts a first input corresponding to the first information and a second input corresponding to the second information; and an inference unit that inputs target first information, which is the first information to be inferred, and target second information, which is the second information corresponding to the target first information, to the model acquired by the acquisition unit, and causes the model to output target third information, which is an inference result for the target first information, thereby generating the target third information in which the target second information is reflected by the attention mechanism.

[0007] According to the present invention, it is possible to perform inference that appropriately reflects information other than the information input as the inference target.

[0008] FIG. 1 is a diagram illustrating an example of a model configuration according to an embodiment. FIG. 2 is a diagram illustrating an example of an attention mechanism according to an embodiment. FIG. 3 is a diagram illustrating an example of a configuration of an information processing device according to an embodiment. FIG. 4 is a diagram illustrating an example of a data storage unit according to an embodiment. FIG. 5 is a diagram illustrating an example of a model information storage unit according to an embodiment. FIG. 6 is a flowchart illustrating an information processing procedure. FIG. 7 is a diagram illustrating a computer that executes an information processing program. FIG. 8 is a diagram illustrating an example of a conventional model configuration. FIG. 9 is a diagram illustrating an example of a conventional attention mechanism. FIG. 10 is a diagram illustrating an example of experimental results. FIG. 11 is a diagram illustrating an example of experimental results. FIG. 12 is a diagram illustrating an example of experimental results. FIG. 13 is a diagram illustrating an example of experimental results. FIG. 14 is a diagram illustrating an example of experimental results. FIG. 15 is a diagram illustrating an example of experimental results. FIG. 16 is a diagram illustrating an example of experimental results.

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0010] [Embodiment] [Overview] Below, we will explain the technologies that form the basis of the model (machine learning model) used for processing by the information processing device 100 according to the embodiment, and then provide an overview of the model used for processing by the information processing device 100 according to the embodiment.

[0011] The Transformer architecture (Transformer model) disclosed in Non-Patent Document 1 has an attention mechanism and is used in a wide range of natural language processing (NLP) applications, such as named entity recognition, sentiment analysis, and machine translation. An overview of a conventional Transformer model will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of the configuration of a conventional model. Specifically, FIG. 8 is a diagram showing an example of the configuration of a conventional Transformer.

[0012] 8 is a diagram showing a conventional model PM1 which is a conventional Transformer. The conventional model PM1 shown in FIG. 8 includes an Input Embedding 141, a Positional Encoding 142, an encoder unit PE1, an Output Embedding 151, a Positional Encoding 152, a decoder unit PD1, a Linear 161, and a Softmax 162.

[0013] The encoder unit PE1 has a Multi-Head Attention 140a, an Add&Norm 143a, a Feed Forward 144, and an Add&Norm 145. The decoder unit PD1 has a Masked Multi-Head Attention 153, an Add&Norm 154, a Multi-Head Attention 150a, an Add&Norm 155a, a Feed Forward 156, and an Add&Norm 157. Since the conventional model PM1 is similar to the conventional Transformer disclosed in Non-Patent Document 1, a detailed description will be omitted, but a brief overview of the attention mechanism of the conventional model PM1 will be given below with reference to FIG. 9.

[0014] Fig. 9 is a diagram showing an example of a conventional attention mechanism. Specifically, Fig. 9 is a diagram showing an example of the attention mechanism (conventional attention mechanism) in a conventional Transformer disclosed in Non-Patent Document 1. In Fig. 9, elements shown as rectangles correspond to processes (layers), and elements shown as rounded rectangles represent tensors (information).

[0015] The conventional attention mechanism PA1 shown in Fig. 9 corresponds to the Multi-Head Attention 140a and the Add&Norm 143a. In the attention mechanism PA1, a tensor D11 denoted as "input" and a tensor D12 denoted as "memory" are used for processing. For example, the tensor D11 corresponds to input information (e.g., the input of the conventional model PM1).

[0016] The tensor D11 is converted into a tensor D13 denoted as "query" by processing of dense 1401a denoted as "Q-dense." For example, dense 1401a is a layer that executes processing that takes tensor D11 as input and outputs tensor D13. Tensor D13 corresponds to Q in the attention mechanism.

[0017] The tensor D12 is converted into a tensor D14 denoted as "key" by processing of dense 1402a denoted as "K-dense." For example, dense 1402a is a layer that executes processing that takes tensor D12 as input and outputs tensor D14. Tensor D14 corresponds to K in the attention mechanism.

[0018] Furthermore, the tensor D12 is converted into a tensor D15 denoted as "value" by processing of dense 1403a denoted as "V-dense." For example, dense 1403a is a layer that executes processing that takes tensor D12 as input and outputs tensor D15. Tensor D15 corresponds to V in the attention mechanism.

[0019] The processing of Matmul 1404a generates a tensor D16 denoted as “logit.” Matmul 1404a is a layer that receives as input a tensor D13 corresponding to Q and a tensor D14 corresponding to K and executes processing to calculate the matrix product of Q and K.

[0020] The processing of the Softmax 1405a generates a tensor D17 denoted as "attention_weight." The Softmax 1405a is a layer that executes processing to input the tensor D16 and output the tensor D17 by normalizing the result of the matrix multiplication calculated by the Matmul 1404a.

[0021] The processing of Matmul 1406a, dense 1407a (denoted as "Output-dense"), and Add&Norm 143a generates a tensor D18 (denoted as "output"). Matmul 1406a receives tensor D17 and tensor D15 corresponding to V as input, and calculates the matrix product of the weight (attention_weight) and V. Dense 1407a receives the output (tensor) of Matmul 1406a as input and executes processing to output a tensor. Add&Norm 143a is a layer that executes residual connection (skip connection) and normalization processing. Note that, while FIG. 9 illustrates the attention mechanism PA1 including Add&Norm 143a, the attention mechanism PA1 does not necessarily include Add&Norm 143; instead, only Multi-Head Attention 140a may be the attention mechanism PA1. In this case, the output of dense 1407a may be used as the output of attention mechanism PA1.

[0022] In the field of NLP, for example, previous models such as PM1 (mentioned above) have achieved tremendous progress primarily through attention-based mechanisms pre-trained on large datasets, such as Bidirectional Encoder Representations from Transformers (BERT), which has effectively expanded the scope of research on NLP. Representative examples of attention-based architectures include ChatGPT, Gemini, and LLaMA2. Traditional attention-based models typically focus on relationships involving one or two objects, such as queries or memory objects, using attention mechanisms such as self-attention and source-target attention. However, real-world interactions often involve more complex relationships involving three or more objects. For example, two related sentences in a Wikipedia article share a common topic. Similarly, a tweet (https: / / x.com / ) can trigger a series of replies related to a specific location, and multiple posts on Reddit (https: / / www.reddit.com / ) can derive from a shared conversational context. Therefore, being able to incorporate these kinds of complex multi-object relationships can further unlock the potential of Transformer-based models.

[0023] Effective representation of multi-object relationships can be achieved by using appropriate tensors (e.g., high-rank tensors), but existing attention-based methods do not utilize appropriate tensors to represent multi-object relationships. For example, the Tensorized Transformer model disclosed in [2] incorporates a specialized self-attention encoder layer known as multi-linear attention and also utilizes the Block-Term Tensor Decomposition (BTD) technique. The conventional technique disclosed in [2] reduces the computational load by compressing the extensive parameter set of multi-headed attention into a set of third-order tensors.

[0024] However, conventional techniques have several limitations. For example, due to constraints imposed by tensor decomposition, it is difficult to process relationships between more than two objects. Furthermore, the query and memory lengths must be equal, which limits their applicability to cases involving a wider variety of object types. Because conventional techniques directly calculate the Transformer output from the attention weight tensor, they are not well suited to obtaining transformations for different types of target objects. Furthermore, because they simultaneously decompose multiple tensors, they can incur higher memory overhead than Tucker decomposition ("Some mathematical notes on three-mode factor analysis.", L. R. Tucker, Psychometrika, 31:279-311, 1966c). Furthermore, conventional techniques face overfitting problems when using more than two core tensors, and accuracy can decrease when using a single tensor. Therefore, there is room for improvement in models with tensor-based attention mechanisms (also known as "attention models").

[0025] Therefore, it is desirable to extend the existing Tensorized Transformer architecture to introduce a mechanism for efficiently integrating attention weights among multi-object relationships. For example, in traditional two-dimensional attention models, queries use keys to determine where to retrieve information from memory, but a multi-dimensional approach is desirable to represent complex real-world phenomena involving multiple object relationships.

[0026] Therefore, the information processing device 100 introduces an object type called a semantic object (also simply referred to as "semantics"). Semantics complements conventional query objects (also simply referred to as "queries") and memory objects (also simply referred to as "memories"). For example, semantics emphasizes the central role of semantics in capturing important aspects of relationships between multiple objects and enables seamless connection between two different objects, a query and a key. This enables the model used by the information processing device 100 to more effectively learn and represent multidimensional relationships, accurately understand co-occurrences between objects, and more appropriately determine which part of the memory to retrieve information from, thereby improving the accuracy of inference processing (prediction accuracy).

[0027] For example, when a series of related sentences are encountered, a common topic can be a semantic. In social networking services (SNSs) such as X, a specific location can also be a semantic, similar to the context of a shared conversation on Reddit. By incorporating these semantics, the model used by the information processing device 100 can effectively represent multiple object relationships. For example, the model used by the information processing device 100 achieves these advances by re-envisioning the current tensorized Transformer framework.

[0028] First, the model used by the information processing device 100 performs Tucker decomposition based on a matrix of the object dimension size. This allows the information processing device 100 to calculate multidimensional attention even if the lengths of vectors corresponding to each object, such as the query, memory, and semantics, are different. For example, the model used by the information processing device 100 improves accuracy by implementing tensor decomposition that aggregates information from the memory and semantics components in a manner that matches the length of the query.

[0029] The model used by the information processing device 100 calculates multidimensional attention between the query, the key (information derived from memory), and semantics. The model used by the information processing device 100 then utilizes this multidimensional attention to learn a transformation from the source (value from memory) to the target (query). For example, the model used by the information processing device 100 optimizes memory usage for two sets of multidimensional attention associated with a maximum of two core tensors by employing an iterative process that averages attention calculations obtained from Tucker decomposition. This significantly reduces memory consumption compared to the conventional technique of Non-Patent Document 2.

[0030] From here, an overview of Tucker decomposition and multi-linear attention, which are technologies used in the model used by the information processing device 100, will be described. First, an overview of Tucker decomposition will be described. Note that since Tucker decomposition itself is a known technology, the following mainly describes points related to the model used by the information processing device 100, and detailed explanations will be omitted as appropriate.

[0031] Tucker decomposition can model the three-dimensional attention weights between different types of objects. Given a three-dimensional tensor as shown in Equation (1), Tucker decomposition can decompose the three-dimensional tensor into a core tensor G (indicated in italics in the equations) and three factor matrices U as shown in Equations (2) to (5) below. (1) , U (2) , U (3) A third-order tensor can be considered as a three-dimensional extension of the concept of a matrix. In the following, we will refer to the elements of a third-order tensor as A d1,d2,d3 In addition, ":" may be used as an index to fix a specific dimension in a tensor, and then a surface consisting of the remaining dimensions may be extracted. In this case, for example, A d1,d2,;: indicates the extraction of a vector in which the first and second dimensions are fixed and the third dimension is open. Note that, for simplicity, the following explanation will be given using a third-order tensor, but tensors are not limited to third-order tensors and can be extended to tensors of higher dimensions than third order.

[0032]

[0033] In equation (1), A (in italics) represents a third-order tensor. The other elements in equation (1) are expressed by the following equations (2) to (5).

[0034]

[0035]

[0036]

[0037]

[0038] Based on the above, the Tucker decomposition is defined as follows (6):

[0039]

[0040] Here, x in equation (6) ｋ denotes the tensor-matrix product along the kth mode.

[0041] Next, we provide an overview of multi-linear attention. An attention mechanism based on multi-linear attention is constructed for a self-attention mechanism that focuses on multi-head attention. First, we assume that the query, key, and value can be mapped to three orthogonal basis vectors, respectively. The dimensions of the factor matrices (denoted as Q, K, and V, respectively) for the query, key, and value are Nxd, where N is the length of the sequence and d is the dimension of the matrix.

[0042] Here, we initialize a trainable tri-diagonal tensor G with rank R. The single-head attention mechanism is defined by Equation (7) below.

[0043]

[0044] In this equation, A(G;Q,K,V) (the term on the left side) represents the attention weight tensor, which stores the attention weights between the query, key, and value as its elements. Here, G acts as the core tensor, and q, k, and v are its indices. Also, Q, K, and V represent the lengths of the query, key, and value, respectively, and the symbol shown in equation (8) represents the cross product.

[0045]

[0046] Q q,: , K. k,: , and V v,: represent column vectors extracted from Q, K, and V, respectively. For example, as in Non-Patent Document 2, it is sometimes assumed that Q = K = V = R. The core tensor G is initialized as shown in the following equation (9).

[0047]

[0048] The core tensor G is a trainable element g computed using a softmax function along the diagonal. r Then, a multi-head mechanism is constructed by applying block-term tensor decomposition, which is called multi-linear attention and can be formulated as follows:

[0049]

[0050] According to Equation (10), M(G;Q,K,V) (the term on the left side) represents the attention output matrix calculated by a linear function directly applying the attention weight tensor A. The core tensor G h (shown in italics in the formula) is the diagonal tensor, and the core tensor G h The number of parameters in is equal to the rank of the core tensor, and h is expressed by the following equation (11).

[0051]

[0052] In equation (10), G is a set of core tensors, and SplitConcat(·) is a function that realizes concatenation after splitting a third-order tensor.O is the parameter matrix, a fully connected layer, and is related to the output of Multi-Linear Attention. q ,W k , and W v are parameter matrices that are learned to adjust the dimensionality of Q, K, and V, respectively, and are shared when constructing multiple core tensors (also simply called "cores") within Multi-Linear Attention.

[0053] Based on the above content, the following describes an overview of the model used by the information processing device 100. Note that explanations of points similar to those described above will be omitted as appropriate.

[0054] First, the model used by the information processing device 100 has a multi-object attention mechanism for more accurately capturing real-world relationships. In a conventional self-attention mechanism, a query shown in the following formula (12), a key from memory shown in the following formula (13), and a value from memory shown in the following formula (14) are used.

[0055]

[0056]

[0057]

[0058] D represents the size of the dimension. For example, in Self-Attention, Q, K, and V are identical, while in Source-Target Attention, Q and K are different because they come from different sources (are derived from information).

[0059] Here, the model used by the information processing device 100 enables the expression of multi-object relationships by introducing additional components (semantics) as shown in the following equation (15), which are different from the existing Q, K, and V components.

[0060]

[0061] For example, in the model used by the information processing device 100, Q, K, and S are all different from one another, while K and V are the same because they come from the same source, memory. This allows the model used by the information processing device 100 to more effectively capture complex dependencies and interactions through an attention mechanism by utilizing semantic information (S). As a result, the information processing device 100 can more accurately model natural relationships in real-world contexts.

[0062] From here, the architecture of the model used by the information processing device 100 will be described with reference to Figs. 1 and 2. Fig. 1 is a diagram showing an example of the configuration of a model according to an embodiment. Fig. 2 is a diagram showing an example of an attention mechanism according to an embodiment. Specifically, Fig. 2 is a diagram showing an example of the configuration of an improved attention mechanism. Fig. 1 is also a diagram showing an example of the configuration of a model having the attention mechanism shown in Fig. 2. Note that explanations of points similar to those of the conventional Transformer described above will be omitted as appropriate.

[0063] 1 is a diagram showing a model M1, which is an example of a model used by the information processing device 100. The model M1 shown in FIG. 1 includes an Input Embedding 141, a Positional Encoding 142, an encoder unit EN1, an Output Embedding 151, a Positional Encoding 152, a decoder unit DC1, a Linear 161, and a Softmax 162.

[0064] The encoder unit EN1 has Multi-Head Attention 140, Add&Norm 143, Feed Forward 144, and Add&Norm 145. The decoder unit DC1 has Masked Multi-Head Attention 153, Add&Norm 154, Multi-Head Attention 150, Add&Norm 155, Feed Forward 156, and Add&Norm 157. Note that model M1 is similar to the conventional model PA1 shown in Figures 8 and 9 except for the configuration of the attention mechanism, so detailed description will be omitted and the attention mechanism (hereinafter also referred to as the "semantic attention mechanism") of model M1 will be described below with reference to Figure 2.

[0065] In Fig. 2, elements shown by rectangles correspond to processes (layers), and elements shown by ellipses represent tensors (information). The attention mechanism AT1, which is an example of a semantic attention mechanism shown in Fig. 2, corresponds to Multi-Head Attention 140 and Add&Norm 143.

[0066] The attention mechanism AT1 uses a tensor D1 denoted as "input," a tensor D2 denoted as "semantics," and a tensor D3 denoted as "memory" for processing. For example, the tensor D1 corresponds to first information, which is one of the input information of the model M1 described later, and the tensor D2 corresponds to second information, which is one of the input information of the model M1 described later. In this way, the second information is input to the model M1 in addition to the first information.

[0067] Tensor D1 is converted into tensor D4, denoted as "query," by processing of dense 1401, denoted as "Q-dense." For example, dense 1401 is a layer that executes processing that takes tensor D1 as input and outputs tensor D4. Tensor D4 corresponds to Q in the attention mechanism.

[0068] Tensor D2 is converted into tensor D5, denoted as "semantics," by processing of dense 1402, denoted as "S-dense." For example, dense 1402 is a layer that executes processing that takes tensor D2 as input and outputs tensor D5. Tensor D2 corresponds to S, which is an input element newly added to the attention mechanism. In this way, the semantic attention mechanism has a configuration corresponding to S.

[0069] Tensor D3 is converted into tensor D6 denoted as "key" by processing of dense 1403 denoted as "K-dense." For example, dense 1403 is a layer that executes processing that takes tensor D3 as input and outputs tensor D6. Tensor D6 corresponds to K in the attention mechanism.

[0070] Furthermore, tensor D3 is converted into tensor D7 denoted as "value" by processing of dense 1404 denoted as "V-dense." For example, dense 1404 is a layer that executes processing that takes tensor D3 as input and outputs tensor D7. Tensor D7 corresponds to V in the attention mechanism.

[0071] A QxKxS tensor is generated by the processing of Tucker decomposition 1405, which is referred to as "Multi-object Attention Tucker Decomposition." The Tucker decomposition 1405 is a layer that executes processing to generate (calculate) a QxKxS tensor using a tensor D4 corresponding to Q, a tensor D5 corresponding to S, and a tensor D6 corresponding to K as input.

[0072] A QxKxS tensor is converted into a QxK tensor by the processing of fusing 1406, which is referred to as "Semantic fusing." The fusing 1406 is a layer that executes processing to input a QxKxS tensor and output a QxK tensor.

[0073] The processing of Matmul 1407, dense 1408 denoted as "Output-dense", and Add&Norm 143 generates a tensor D8 denoted as "output". Matmul 1407 takes as input the QxK tensor generated by fusing 1406 and tensor D7 corresponding to V, and calculates the matrix product of the weight (attention_weight) and V. For example, Matmul 1407 generates a QxD tensor. dense 1408 is a layer that executes processing to output a tensor using the output (QxD tensor) of Matmul 1407 as input. For example, dense 1408 generates a Qx768 tensor.

[0074] The Add&Norm 143 is a layer that performs residual connection (skip connection) and normalization processing. Note that, in FIG. 2, the attention mechanism AT1 is shown including the Add&Norm 143, but the attention mechanism AT1 does not necessarily include the Add&Norm 143, and only the Multi-Head Attention 140 may be the attention mechanism AT1. In this case, the output of the dense 1408 may be used as the output of the attention mechanism AT1.

[0075] As mentioned above, the attention mechanism (semantic attention mechanism) of model M1 shown in Figure 2 differs from conventional attention mechanisms in that it accepts not only query and memory but also semantics as inputs. Here, each of these inputs first passes through a dense network. q is the query weight matrix, W s is the semantics weight matrix, W k is the key weight matrix, W vwhere m and s represent the value weight matrices. Model M1 calculates a 3D attention tensor A between the query matrix Q, semantic matrix S, and key matrix K using multi-object attention with Tucker decomposition. To incorporate these 3D attention weight matrices into the value matrix V, model M1 converts the 3D attention tensor A (shown in italics in Figure 2) into a 2D matrix A through semantic fusion. This process involves combining qxk matrices along the semantic direction. Model M1 then performs Matmul (matrix multiplication) between this 2D matrix A and the value matrix V. Model M1 then applies Add&Norm, i.e., addition and normalization, between the updated values ​​and the original query. Finally, model M1 generates an output O (tensor D8) through this process.

[0076] With the configuration of the model M1 described above, the information processing device 100 calculates the attention weights between Q, K, and S using multi-object attention Tucker decomposition, as in the process of Tucker decomposition 1405. The obtained three-dimensional attention weights are reflected in the output of the model M1 via semantic-fused 2D attention, as in the process of fusing 1406.

[0077] For example, in the technique disclosed in Non-Patent Document 2, the lengths of the matrices Q, K, and V are restricted to be the same. However, the relationship between multidimensional objects in the real world naturally includes objects of different lengths. Therefore, in model M1, a column vector Q of length D extracted from Q, K, and S is used. q,: , K. k,: , S s,: In order to perform multi-object attention Tucker decomposition using the same method as in the Tucker decomposition 1405 process, the following equation (16) obtained by modifying equation (7) is used.

[0078]

[0079] For example, the length of a column vector is always the dimension size, and is denoted as D respectively so that they all have the same length. Here, the symbol "·" in the formula represents the inner product of two tensors. The size of the core tensor G is set so that the length of Q remains Q, and the center of the relationship between Q, K, and S can be arranged around Q. For example, when calculating the outer product of three matrices and calculating the sum over D dimensions with respect to Q, a row vector Q of length Q :,i is selected. In Equation (16), when calculating the outer product of Q :,i , K k,: , and S s,: and calculating the sum over D dimensions with respect to Q, the first, fourth, and sixth dimensions are activated. In a single core tensor G, there is mainly a trainable weight vector g of length D on the diagonal, which is initialized by the following Equation (17).

[0080]

[0081] Each element of g is represented as g d with respect to d represented by the following Equation (18), and is calculated using the softmax function along its diagonal, and is represented by the following Equation (19).

[0082]

[0083]

[0084] For example, if D < Q, the elements that become D < q become zero. Also, if D ≥ Q, the length of g becomes the same as the length of Q, that is, D = Q.

[0085] In the technique disclosed in Non-Patent Document 2, a technique known as "split&concat" is adopted and used to convert multi-dimensional attention into a two-dimensional form. In this technique, the three-dimensional attention obtained from Attention Tucker Decomposition is split along the V-axis, and they are concatenated to create a two-dimensional matrix of size R Qx(S・K) . Then, this two-dimensional matrix has a size of R through a Linear layer QxDHowever, this method does not fully utilize the advantages of the Transformer, especially when dealing with the transformation from a source object to a target object of a different type. This is likely because the traditional Transformer cannot effectively transform V using the attention output and seamlessly integrate it with Q.

[0086] To address these limitations, model M1 effectively transforms the S-axis of the multidimensional attention tensor into a two-dimensional matrix. This semantic fusion function (e.g., the function used in the fusing 1406 process) is formulated by re-expressing Equation (16) as Equation (20) below.

[0087]

[0088] As a result, in model M1, two-dimensional attention between Q and K is calculated based on the semantically rich three-dimensional attention tensor. q,k By calculating the inner product (or Matmul) of V and V, it is possible to directly capture the transformation caused by the attention mechanism in V itself. Also, in model M1, the residual connection is the parameter matrix W O and Q, and produces the output O (tensor D8) as shown in equation (21) below.

[0089]

[0090] In addition, the technique disclosed in Non-Patent Document 2 introduces tensor decomposition with multiple cores to implement multidimensional attention (multi-core attention), but overfitting and a decrease in accuracy occur when the number of cores exceeds three. Furthermore, despite efforts at low-rank approximation, the technique disclosed in Non-Patent Document 2 faces computational challenges in consuming large amounts of memory in experiments.

[0091] To solve this problem, the information processing device 100 employs model M1 to calculate multidimensional attention using multiple core tensors and averages them to construct the final multidimensional attention weights. This allows the information processing device 100 to significantly improve memory efficiency when predicting multidimensional attention. This is achieved, for example, by the information processing device 100 dynamically summing three-dimensional attention tensors on the fly using model M1. In this case, there is no need to calculate and store the tensors individually, and then fold them along the S-axis during calculation. This method allows the information processing device 100 to improve memory utilization. Furthermore, the information processing device 100 can remove the Q = K = L constraint by prioritizing tensor decomposition centered on D using model M1.

[0092] These functions enable the information processing device 100 to improve accuracy even when using a single core, reduce the possibility of overfitting or memory problems, and provide consistently robust performance even when configured with three or more cores. For example, the information processing device 100 performs calculations iteratively in the range of 1≦n≦(N−1) using an equation such as that shown in the following formula (22).

[0093]

[0094] where n is the index of the N core tensors, and G n q,j,l is the n-th core tensor. After each iteration, the information processing device 100 n q,k is discarded to save memory. q,k is calculated by averaging as shown in the following equation (23).

[0095]

[0096] By using the model M1 having the above-described configuration, the information processing device 100 can perform inference that appropriately reflects information other than the information input as the inference target. In the above example, the encoder unit EN1 has the attention mechanism AT1, which is a semantic attention mechanism. However, the semantic attention mechanism may be provided in the decoder unit DC1. In this way, the other attention mechanisms in the model M1 may also have a configuration similar to that of the attention mechanism AT1. Since the configurations are substantially the same as those described for the attention mechanism AT1, detailed description will be omitted. For example, the attention mechanism AT2 may have a configuration similar to that of the attention mechanism AT1. For example, if the attention mechanism AT2 has a configuration similar to that of a conventional attention mechanism, the Multi-Head Attention 150 may have a configuration similar to that of the Multi-Head Attention 150a, and the Add&Norm 155 may have a configuration similar to that of the Add&Norm 155a.

[0097] Semantic attention mechanisms such as attention mechanism AT1 excel in their ability to naturally process multidimensional real-world relationships and may be combined with pre-trained language models based on conventional two-dimensional (2D) attention mechanisms, such as the currently widely used BERT-based model. For example, integration with a pre-trained language model may be achieved by providing semantic information (e.g., second information) corresponding to the output from the pre-trained language model as input to a model with a semantic attention mechanism such as model M1. This allows models with semantic attention mechanisms such as model M1 to be used with pre-trained models and be expandable to various applications. In other words, models with semantic attention mechanisms such as model M1 are applicable to a variety of applications.

[0098] Furthermore, any manner can be adopted for training a model with a semantic attention mechanism such as model M1. For example, a masked language model (MLM) and next sentence prediction (NSP) may be pre-trained using an input (e.g., first information), a memory, and semantic information (e.g., second information). In this case, the model is optimized using the sum of the cross-entropy losses of the MLM and NSP. Furthermore, for fine-tuning response selection, for example, the model with a semantic attention mechanism uses a hidden vector Ocls extracted from the first cls token in the output O (e.g., tensor D8). Then, a match score between the current speech context and the response is calculated using a single-layer neural network. For example, the match score is expressed as in the following equation (24):

[0099]

[0100] For example, W in Equation (24) represents a trainable parameter. In the case of question-answering, the model can be trained by treating the question as the current speech context and the answer as the response. The weights of the model are updated using a cross-entropy loss function. Note that the above is merely an example, and a model with a semantic attention mechanism like model M1 (hereinafter also referred to as a "tensorized attention model") may be generated by any learning method, not limited to the above.

[0101] [Configuration of Information Processing Apparatus] Next, a configuration of the information processing apparatus 100, which is an example of an information processing apparatus that executes information processing according to the embodiment, will be described. Fig. 3 is a diagram showing an example of the configuration of the information processing apparatus 100 according to the embodiment.

[0102] 3, the information processing device 100 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also include an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator of the information processing device 100, a display unit (e.g., a liquid crystal display, etc.) that displays information, and an output unit such as an audio output unit (e.g., a speaker, etc.) that outputs information aloud.

[0103] The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to a predetermined network such as the Internet via a wired or wireless connection, and transmits and receives information to and from other information processing devices such as terminal devices used by users who receive the inference service provided by the information processing device 100.

[0104] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 3 , the storage unit 120 according to the embodiment includes a data storage unit 121 and a model information storage unit 122.

[0105] The data storage unit 121 according to the embodiment stores data used for processing. For example, the data storage unit 121 stores data used for learning (also referred to as "learning data"). FIG. 4 is a diagram illustrating an example of the data storage unit according to the embodiment. In the example of FIG. 4, the data storage unit 121 includes items such as "data ID," "first information," "second information," and "third information."

[0106] "Data ID" indicates identification information for identifying each data. "First information" indicates first information included in data identified by the data ID. The first information is information corresponding to a first object. The first information is used as input to a machine learning model (model) that executes inference processing.

[0107] For example, the first information is information corresponding to a first object that triggers the execution of the inference process. For example, the first information is information that requests a response. For example, the first information is information entered by a person in an internet service (also called an "internet service") such as a social networking site or a bulletin board. For example, the first information is information indicating a question, which is an example of a first object. For example, the first information is information indicating a comment, which is an example of a first object. For example, the first information may be information generated from a person's speech.

[0108] Furthermore, "second information" refers to second information included in the data identified by the data ID. The second information is information corresponding to a second object. The second information is used as input to a machine learning model (model) that executes inference processing.

[0109] For example, the second information is information corresponding to a second object related to the first object. For example, the second information is information indicating a specific subject related to the corresponding first information. For example, the second information is information indicating a second object related to a request for a response. For example, the second information is information related to the content of a question. For example, the second information is information indicating the topic (subject) of a question, which is an example of a second object. For example, the second information may be the title of a thread in an Internet service such as a social networking site or a bulletin board, in which the first information was entered (posted). Note that the second information may be any semantic information that is semantically related to the first information. For example, the second information may be any information that enables more accurate inference of the meaning of the first information.

[0110] Furthermore, "third information" refers to third information included in the data identified by the data ID. The third information is information corresponding to a third object. The third information is used as correct answer information (correct answer label) when training a machine learning model (model).

[0111] For example, the third information is information corresponding to a third object that is a reaction (response) to the first information. For example, the third information is information that indicates a third object in response to a request for a response. For example, the third information is information that indicates a response, which is an example of a third object. For example, the third information is information that indicates an answer to a question, which is an example of a third object. For example, the third information may be an answer to the first information, which is a question in an Internet service. Note that the third information may be any information that is a response to the first information. The third information may be a reply to the first information, etc.

[0112] 4 shows that data identified by a data ID "DID1" (data DID1) includes first information FD1, second information SD1, third information TD1, etc. For example, data DID1 includes first information FD1 indicating a question, second information SD1 indicating a topic corresponding to the question, third information TD1 indicating a response to the question, etc.

[0113] The data storage unit 121 may store various types of information depending on the purpose, without being limited to the above. For example, the data storage unit 121 may store information to be inferred. In this case, the data storage unit 121 may have an item indicating whether the information is information used for learning (learning data) or information to be inferred (inference target data), and may store information indicating the type, such as "for learning" or "for inference," as information corresponding to the item. For example, the data storage unit 121 may store target first information and target second information to be inferred. In this case, the data storage unit 121 may store information such as a "-" (hyphen) in the third information item corresponding to the target first information and target second information to be inferred. Furthermore, after performing an inference process using the target first information and target second information to be inferred, the data storage unit 121 may register the target third information generated by the inference process in the third information item corresponding to the target first information and target second information.

[0114] The model information storage unit 122 according to the embodiment stores information related to a model. For example, the model information storage unit 122 stores information (model data) indicating the structure of a model (network). FIG. 5 is a diagram illustrating an example of the model information storage unit according to the embodiment. In the example shown in FIG. 5, the model information storage unit 122 includes items such as "model ID," "purpose," and "model data."

[0115] "Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model Data" indicates the data of the model. Figure 5 shows an example in which conceptual information such as "MDT1" is stored in "Model Data", but in reality, various information constituting the model is included, such as information about the network included in the model, functions, and parameter information of the model learned by the learning process.

[0116] In the example shown in FIG. 5, the model (model M1) identified by the model ID "M1" indicates that its use is "response generation." This indicates that the model M1 is a model used for response generation. This also indicates that the model data of the model M1 is model data MDT1.

[0117] The model information storage unit 122 is not limited to the above, and may store various types of information depending on the purpose. For example, the model information storage unit 122 may store multiple models. The model information storage unit 122 may store multiple models generated for multiple applications.

[0118] Returning to Fig. 3 , the explanation will be continued. The control unit 130 is realized, for example, by a processor such as a CPU (Central Processing Unit) executing a program (for example, an information processing program such as an inference processing program) stored inside the information processing device 100 using a RAM or the like as a work area. The control unit 130 is also realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). As shown in Fig. 3 , the control unit 130 has an acquisition unit 131, an inference unit 132, a learning unit 133, and a provision unit 134.

[0119] The acquisition unit 131 acquires information. The acquisition unit 131 acquires information from an external information processing device. The acquisition unit 131 acquires information from a user's terminal device. The acquisition unit 131 acquires information from the storage unit 120. The acquisition unit 131 acquires information from the data storage unit 121 and the model information storage unit 122.

[0120] The acquisition unit 131 acquires a model that receives first information corresponding to a first object and second information corresponding to a second object and added to the first information as input, and outputs third information indicating an inference result for the first information, the model including an attention mechanism that accepts a first input corresponding to the first information and a second input corresponding to the second information. The acquisition unit 131 acquires a model that includes an attention mechanism that can process data of different dimensional sizes by Tucker decomposition.

[0121] The acquisition unit 131 acquires a model to be used in the inference process from the model information storage unit 122. For example, the acquisition unit 131 acquires a model M1 from the model information storage unit 122. The acquisition unit 131 acquires data to be used in the inference process from the data storage unit 121. The acquisition unit 131 acquires first information and second information to be used in the inference process from the data storage unit 121. The acquisition unit 131 may acquire first information (target first information) from a user who requests the inference process. The acquisition unit 131 may acquire second information (target second information) from the user who requests the inference process. Furthermore, when the information processing device 100 performs learning, the acquisition unit 131 acquires learning data to be used in learning the model from the data storage unit 121.

[0122] The inference unit 132 executes an inference process for inferring information. The inference unit 132 executes the inference process based on information from an external information processing device and information stored in the storage unit 120. The inference unit 132 executes the inference process based on information stored in the data storage unit 121 and the model information storage unit 122.

[0123] The inference unit 132 executes an inference process based on the information acquired by the acquisition unit 131. The inference unit 132 executes the inference process using a machine learning model (model). For example, the inference unit 132 executes the inference process using a model that includes an attention mechanism that accepts a first input corresponding to the first information and a second input corresponding to the second information, receives the first information and the second information as inputs, and outputs third information that indicates an inference result for the first information.

[0124] The inference unit 132 inputs target first information, which is first information to be inferred, and target second information, which is second information corresponding to the target first information, into the model acquired by the acquisition unit 131, and causes the model to output target third information, which is an inference result for the target first information, thereby generating target third information in which the target second information is reflected by an attention mechanism. The inference unit 132 inputs the target first information and the target second information into the model, and generates the target third information by having the model output the target third information by processing data of different dimensional sizes in the attention mechanism.

[0125] The inference unit 132 performs inference processing using a model such as model M1. For example, the inference unit 132 generates a response corresponding to a question using model M1, which is a response generation model. The inference unit 132 inputs a question, which is first information to be inferred, and the topic of the question, which is second information corresponding to the question, to model M1, and causes model M1 to output an answer (response), which is an inference result for the question, thereby generating a response in which the topic is reflected by an attention mechanism. When multiple models are registered in the model information storage unit 122, the inference unit 132 selects a model corresponding to the inference processing from the multiple models registered in the model information storage unit 122, and performs the inference processing using the selected model.

[0126] The learning unit 133 executes a learning process to learn a model (machine learning model). The learning unit 133 executes the learning process based on information from an external information processing device and information stored in the storage unit 120. The learning unit 133 executes the learning process based on learning data stored in the data storage unit 121. The learning unit 133 stores the model generated by the learning process in the model information storage unit 122.

[0127] The learning unit 133 executes a learning process based on the information acquired by the acquisition unit 131. The learning unit 133 learns a model using various machine learning techniques. For example, the learning unit 133 learns parameters of the model. The learning unit 133 learns a model using various machine learning techniques. For example, the learning unit 133 learns parameters of model M1.

[0128] The learning unit 133 performs a learning process using the learning data (teacher data) stored in the data storage unit 121. The learning unit 133 learns a model to be used for response generation through the learning process using the learning data. For example, the learning unit 133 learns a model M1.

[0129] The learning unit 133 performs a learning process when first information and second information are input so that the information output by the model M1 approaches third information (correct information) corresponding to the first information and second information. For example, when using the learning data DID1 shown in FIG. 4 , the learning unit 133 learns the model M1 so that the model M1 to which the first information FD1 and the second information SD1 are input outputs third information TD1 (correct information). In this case, the learning unit 133 performs a learning process using a method such as backpropagation (backpropagation method) so that the information output by the model M1 to which the first information FD1 and the second information SD1 of the learning data DID1 are input approaches the third information TD1 of the learning data DID1.

[0130] The learning method used by the learning unit 133 is not limited to the above, and any learning method can be used as long as a desired model such as model M1 can be learned. Furthermore, any structure can be used as the network structure (internal structure) of a model such as model M1 as long as desired information can be output in response to input. For example, a model (network) in the form of a neural network (NN: Neural Network) such as a deep neural network (DNN) may be used as a model such as model M1. Note that any type of model (function) may be used as a model such as model M1 as long as desired information can be output.

[0131] Note that if the information processing device 100 does not perform a learning process to learn a model, the information processing device 100 may not have the learning unit 133. That is, models such as model M1 may be learned (generated) by the information processing device 100, or may be learned (generated) by a device (machine learning device) other than the information processing device 100. For example, when a machine learning device learns a model such as model M1, the information processing device 100 may acquire (receive) a model such as model M1 from the machine learning device and perform processing such as inference processing using the model such as model M1. Furthermore, the information processing device 100 may receive a pre-trained model (base model) from the machine learning device, and the learning unit 133 may generate a model such as model M1 by fine-tuning the base model.

[0132] The providing unit 134 provides information. The providing unit 134 provides information to an external information processing device. The providing unit 134 transmits information to an external information processing device. For example, the providing unit 134 transmits information to another information processing device such as a user's terminal device. The providing unit 134 transmits information stored in the memory unit 120. The providing unit 134 transmits information inferred by the inference unit 132. The providing unit 134 provides the inference result to a user who requests the inference process. For example, the providing unit 134 transmits the inference result to the user's terminal device that is the sender of the target first information.

[0133] For example, the providing unit 134 transmits a response (answer) to a question acquired from a user to the user's terminal device. For example, the providing unit 134 transmits the response to a question acquired from a user to the user's terminal device together with second information (semantic information, etc.) used in the inference process. For example, the providing unit 134 transmits a response (answer) to a question entered by a user in an Internet service to an external information processing device such as a server device that provides the Internet service. The providing unit 134 may transmit the response (answer) to a question entered by a user in the Internet service to the user's terminal device via the Internet service. For example, when multiple models are registered in the model information storage unit 122, the providing unit 134 may provide the user with information indicating which model, from among the multiple models registered in the model information storage unit 122, the inference unit 132 used in the inference process.

[0134] [Inference Processing] Next, a description will be given of the inference processing performed by the information processing device 100. Fig. 6 is a flowchart showing the procedure of the inference processing. The flowchart in Fig. 6 starts, for example, when an input is made to instruct the start of the inference processing.

[0135] First, the information processing device 100 acquires a model including an attention mechanism that accepts a first input corresponding to the first information and a second input corresponding to the second information (step S101). For example, the information processing device 100 acquires a model M1, which is a response generation model, from the model information storage unit 122.

[0136] The information processing device 100 inputs the first target information and the second target information into the model and causes the model to output the third target information, thereby generating the third target information in which the second target information is reflected by the attention mechanism (step S102). For example, the information processing device 100 inputs the first target information to be inferred and the second target information into the model M1, and generates the third target information (response) output by the model M1 as an inference result (response) to the first target information.

[0137] [Experimental Results] Here, the experimental results using the above-described configuration will be shown.

[0138] First, we will explain the datasets used in the experiments. In the experiments described below, we used the NFL, Politics, and TweetQA datasets. The two datasets, NFL and Politics, were collected by sampling Reddit posts from the respective communities from September 2018 to February 2019 to evaluate response generation (also known as "response selection"). The datasets can be accessed, for example, through BigQuery (https: / / bigquery.cloud.google.com / dataset / fh-bigquery).

[0139] In the experiments described below, we target active users (speakers) and pair their comments with replies, splitting the data into training and test sets. Comments and replies discussed between users form "threaded dialogues." The NFL dataset contains 230,060 dialogues (average 4.2 utterances, 56.3 words) in the training set and 13,765 dialogues (average 4.2 utterances, 57.6 words) in the test set. The Politics dataset contains 290,020 dialogues (average 4.8 utterances, 81.1 words) in the training set and 19,040 dialogues (average 4.9 utterances, 81.5 words) in the test set. The response selection task aims to infer (identify) the final utterance (response) based on the context of the utterances leading up to that response.

[0140] In our evaluation, we focused on the relationships between the following three objects (1) to (3), which encompass key dimensions observed in natural dialogues. (1) "Full Dialogue History": This is derived from the word embedding stream of the current dialogue and serves as Q. We used the BERT-based tokenizer in our experiments. (2) "Current Dialogue Context": This is created by applying a Gated Recurrent Unit (GRU) to the word embedding streams of the utterance context and its responses. The hidden embedding streams for the utterance context and its responses are concatenated and partitioned before concatenating the GRU stream to ensure an unbiased representation of the context. This is represented as K or V. (3) "Topics Discussed in the Dialogue": This is obtained from the word embedding stream of the dialogue's subject title and is represented as S.

[0141] The dialogue length for the NFL dataset was set to 70 and the title length to 35, while the dialogue length for the Politics dataset was set to 180 and 60, respectively, covering 95% of the dialogue and titles in each dataset. The NFL and Politics datasets have significantly different subject and data distributions. The Politics dataset tends to have longer utterances and titles than the NFL dataset, resulting in larger tensor sizes. This is due to the increased length of context, responses, and meaning. Using the NFL and Politics datasets, we can evaluate the superiority of models with semantic attention mechanisms (tensorized attention models) against datasets with different characteristics.

[0142] In the experiments described below, we also conducted evaluations on the TweetQA dataset ("TWEETQA: A Social Media Focused Question Answering Dataset", Wenhan Xiong et al., 2019). This is a large-scale dataset designed for automatic question answering, allowing abstract answers to social media content. It consists of question-passage-answer triples. Passages are tweets used by journalists in news articles, meaning they contain useful information. The dataset contains 10,692 training triples and 1,979 test triples. The average question length is 6.95, and the average answer length is 2.45.

[0143] We prepared three object relations: (1) the topic of the entire passage, derived from the word embedding stream of the passage and functioning as Q; (2) the context of the passage-answer pair, created by applying GRU to the word embedding streams of the passage and answer, represented as K or V; and (3) the question assigned to the passage, obtained from the word embedding stream of the question, represented as S. We set the length of the passage-answer pair to 48 and the length of the question to 15. These settings cover 95% of the passage-answer pairs and questions in the dataset.

[0144] We utilize specific techniques and internal parameter configurations for comparative evaluation with a model that has a semantic attention mechanism (tensorized attention model). In the following experiments, we integrate the tensorized attention model into a BERT implementation and replace its attention layer (e.g., attention mechanism), thereby directly comparing the tensorized attention model with the BERT model. By comparing the BERT model before integrating the tensorized attention model with the tensorized attention model, we can effectively isolate and analyze the differences in multidimensional attention. This allows us to clearly and comprehensively verify the effectiveness of the tensorized attention model in comparison with the standard Transformer encoder. To ensure fair evaluation, we use two variants of BERT. One is denoted as "BERT(CR)" and uses "the entire dialogue history" for NFL and Politics, and "passage-answer pairs" for TweetQA as inputs as queries, keys, and values. The other model, designated "BERT(SCR)," uses the concatenation of the topic of conversation and the entire conversation history for NFL and Politics, and the concatenation of the question and the passage-answer pair for TweetQA. For comparison, we also introduce two variants with similar configurations: "Tensorized Transformer(CR)" and "Tensorized Transformer(SCR)." All models use 12 Transformer encoder layers. We also present evaluation results for varying the number of layers below.

[0145] The evaluation index in the following experiments is Recall 10 @k("R 10 @k") is used. 10 In @k, given 10 responses, the evaluation measures whether the relevant response ranks among the top k candidates for answer selection (question answering). This metric employs nine responses randomly sampled from the test dataset as negatives, thereby considering the importance of both top-1 and bottom-up predictions in various applications.

[0146] The size of the word embeddings was set to 768, following "BERT: Pre-training of deep bidirectional transformers for language understanding," Jacob Devlin et al., 2019. The learning rate was 1x10 -5 The optimizer was set to a beta of 0.9 and 0.999, respectively, and an epsilon of 1 x 10, following the evaluation settings outlined in "Decoupled weight decay regularization.", Ilya Loshchilov et al., 2019. -8 The AdamW optimizer was used.

[0147] For the NFL and Politics datasets, 20 epochs for pre-training and 15 epochs for fine-tuning achieved sufficient convergence. For the TweetQA dataset, 100 epochs for pre-training followed by 20 epochs for fine-tuning were used.

[0148] For accuracy comparison, we checked reproducibility with five random seeds. The dimension size D of Q, K, V, and S for the NFL dataset was set to 192, and the dimension size D for the Politics dataset was set to 160. The dimension size D of Q, K, V, and S for the TweetQA dataset was also set to 160. The batch size was set to 96 for all methods on all datasets.

[0149] The Tensorized Transformer tends to perform poorly on a single core and is hindered by memory constraints when using three or more cores, so we utilized two cores for optimal performance. For the Tensorized Attention model, we set the number of cores to three for the NFL dataset and 20 for the Politics dataset. Details on the settings for the number of cores, dimension size, and batch size are described below. The hardware used for the evaluation was an NVIDIA A100 GPU with 80GB of memory.

[0150] The experimental results will be explained below using Figs. 10 to 16. Figs. 10 to 16 are diagrams showing examples of experimental results. First, the results of Fig. 10 will be explained. Fig. 10 shows the results of training a model from scratch without using a pre-trained language model. "TTrans" in Fig. 10 is an abbreviation for Tensorized Transformer.

[0151] First, as shown in Figure 10, BERT(SCR) shows higher accuracy than BERT(CR) on the NFL and TweetQA datasets, but the opposite is true on the Politics dataset. These results suggest that while semantic information can be effective in improving accuracy, current attention models struggle to fully exploit the potential benefits of different object types.

[0152] Second, the Tensorized Transformer (CR) and the Tensorized Transformer (SCR) show slight improvements or slight degradation compared to BERT on all datasets, which is consistent with expected results since the primary goal of the Tensorized Transformer is to improve the memory efficiency of single-object self-attention, limiting its ability to achieve significant accuracy gains in multi-object scenarios.

[0153] Additionally, tensorized attention models achieve extremely high accuracy on all datasets trained from scratch. This highlights the importance of using multi-object tensors to naturally represent real-world relationships. Tensorized attention models significantly improve prediction accuracy due to their ability to aggregate information from various objects around Q and facilitate the transformation between source and target objects. For example, tensorized attention models observed on the TweetQA dataset achieve significantly higher accuracy compared to other methods (including BERT (SCR) and Tensorized Transformer (SCR)), particularly compared to improvements observed on other datasets. For example, tensorized attention models consider the stronger association between questions and passage-answer pairs, as opposed to the relationship between titles and utterances in the NFL and Politics datasets. Furthermore, with small datasets like TweetQA, effective utilization of multi-object relationships is critical and necessary to make the most of a limited amount of observations.

[0154] In terms of the number of model parameters, the Tensorized Attention model increases slightly compared to BERT, adding dense networks and core tensor weights, while the Tensorized Transformer significantly reduces the length-dependent W associated with KxV, especially on the Politics dataset. o Due to the values, the number of parameters has increased significantly.

[0155] Next, for ablation, we consider the "without Semantic Fusing" method, which uses the same input configuration as the tensorized attention model. For example, the query is used as the dialogue history, the key and value are used as the current utterance context, and the semantic is used as the dialogue title. However, "without Semantic Fusing" uses split&concat to directly calculate the output of the transformer via the attention weight tensor. We also consider "without Query Aligned," which shares the tensorized attention model method but with a core tensor size of R. DxDxD and R QxDxD (See equation (16)). This means there is no alignment around Q.

[0156] Figure 11 shows the results for ablation on all datasets. "Without Semantic Fusing" performs poorly compared to the tensorized attention model, as it does not learn the transformation from source objects to distinct target objects using attention weight tensors. "Without Query Aligned" also performs less accurately than the tensorized attention model, which is due to the tensorized attention model's ability to aggregate information from other objects around Q through tensor decomposition. This is consistent with the process by which the Transformer updates Q and provides an explanation for the observed performance difference. In particular, the results show that the query-aligned approach is successful in improving accuracy by focusing on Q-centered alignment in learning core tensors. This is because the explicit token embedding of the query can explicitly capture the latent meaning derived from the correlations between q, k, and v.

[0157] In our experiments, we present a visual example of the attention weights calculated for both the tensorized attention model and BERT(SCR), the second-best method in the comparison, on the NFL dataset in Figure 12. In the visualization, the shade of the hatching represents the level of attention weight, with darker hatching indicating high attention, medium hatching indicating neutral, and lighter hatching indicating low attention. That is, in Figure 12, darker hatching indicates higher attention and lighter hatching indicates lower attention.

[0158] In the tensorized attention model, the x-axis represents the context of the utterance and the y-axis represents the response. BERT (SCR) plots the semantics and context of the utterance on the x-axis. To improve readability, only specific words on the x-axis and y-axis are labeled. In the example in Figure 12, the semantics is "Marcus Peters intercepted Mahomes with 1:13 remaining," the context is "He's a rookie. The sport is up and there's a turnover," and the response is "I can't believe Donovan Mitchell lost Lottery for the second year in a row."

[0159] Result RS1 in Fig. 12 shows the sliced ​​qxk matrix corresponding to the semantic token "intercept" by the tensorized attention model. Furthermore, result RS2 in Fig. 12 shows the sum of the qxk matrix for all semantic tokens by the tensorized attention model (see Eq. (22)).

[0160] In result RS1 in Figure 12, the tensorized attention model effectively captures the relationships between words such as "intercept" in semantics, "rookie" and "turnover" in the utterance context, and "roty (rookie of the year)" in the response. These relationships are important because they relate to the basic intercept and turnover plays for rookies. In result RS2 in Figure 12, the tensorized attention model retains these relationships even after summing weights across semantic tokens. In contrast, as shown in result RS3 in Figure 12, BERT(SCR) fails to identify such relationships between three different object types.

[0161] Extracting context from text is challenging in the NFL dataset due to the frequent use of casual language and fragmented sentences. Approaches such as BERT reduce these multidimensional relationships to two dimensions by simply prepending semantics to the text. As a result, they may not accurately capture the simultaneous relationships between semantics, context, and responses. In this example, approaches such as BERT may not be able to effectively highlight the central topics of Mahomes' rookie season or the important attributes of NFL rookies. On the other hand, tensorized attention models can effectively capture the relationships between semantics, context, and responses. Tensorized attention models use Tucker decomposition and multi-core tensors to efficiently compute attention weights across diverse object types. Evaluations show that tensorized attention models improve accuracy over traditional Transformers and previous tensorized Transformers.

[0162] We now present details of other evaluations, including combined evaluation of pre-trained models, performance of tensorized attention models with different batch sizes and dimensions, results of tensorized attention models with different tensor decomposition cores, parameter size analysis, and computational performance details.

[0163] In the experiments using the pre-trained model described above, we adopted the following three approaches: (1) BERT with 12 layers. P The output of SCR is integrated with the Tensorized Attention Model (TAM), creating a hybrid model, which is then used for BERT. P (2) The output of BERT(CR) or BERT(SCR) is used as the input of BERT(CR) or BERT(SCR), and the result is labeled as BERT(SCR)-TAM. P (CR)-BERT(CR) or BERT P (3) A single 12-layer BERT. P We compared the model with (SCR). P The term "" refers to the pre-trained BERTbase model (12 layers), while the terms "BERT" or "TAM" refer to the 4-layer untrained model. The choice of 4 layers is due to its superior performance and computational efficiency compared to the 12-layer model.

[0164] In experiments using the pre-trained BERT-base model, BERT P (SCR)-TAM was pre-trained for 6 epochs on the NFL and Politics datasets, and fine-tuned for 6 epochs on both datasets. P All methods except (SCR)-TAM required 15 epochs for fine-tuning on both datasets, which was sufficient for model convergence.

[0165] Figure 13 shows the results when initializing a pre-trained language model: BERT P (SCR)-BERT(SCR) is superior to BERT in terms of accuracy. P (CR)-BERT(CR) and BERT P (SCR). P (SCR)-TAM is BERT P(SCR)-BERT(SCR) achieved higher accuracy. These results demonstrate the benefit of incorporating semantic information when combining a pre-trained traditional language model with a tensorized attention model (TAM).

[0166] Figure 14 shows the performance of the Tensorized Attention Model (TAM) when varying batch size and dimension size. Based on the information in Figure 14 and that in Figure 10, it is clear that the accuracy is consistently high regardless of batch size or dimension size, outperforming other methods and demonstrating stability. This stability is due to the flexibility of the model to utilize low-rank D for computation within the core tensor rather than being limited by query or memory length.

[0167] The results for the Tensorized Attention Model (TAM) are shown in Figure 15, which plots the number of cores used for tensor decomposition. As mentioned above, the Tensorized Attention Model uses Tucker decomposition across multiple cores for efficient memory utilization. As a result, the Tensorized Attention Model can efficiently utilize up to 20 cores for tensor decomposition, while the Tensorized Transformer only uses up to two cores in the evaluation setup. For example, we observed that the optimal number of cores depends on the dataset. For example, the Tensorized Attention Model achieves higher accuracy when using three cores for the NFL dataset and 20 cores for the Politics dataset. Furthermore, the Tensorized Attention Model is significantly more efficient than the Tensorized Transformer, as shown in Figure 10. By introducing Tucker decomposition into the Tensorized Attention Model, increasing the number of cores barely increases the number of parameters, improving memory efficiency.

[0168] In our experiments, we conducted an ablation study using the NFL and Politics datasets, varying the number of layers in BERT(SCR) and TAM. The results are shown in Figure 16. For BERT(SCR), we set the parameters to 21 layers, the word embedding size to 768, and the batch size to 128. These settings were set to the maximum values ​​for the hardware configuration described above. For TAM, we set the parameters to 12 layers, the word embedding size to 768, the batch size to 96, and the dimensionality to 192. These specifications were constrained by available memory limitations. Here, we found that increasing the parameter size of BERT(SCR) beyond the 12 layers of TAM (setting the number of layers of BERT(SCR) to 15 or more in Figure 16) consistently outperformed BERT(SCR) in accuracy. These results suggest that TAM's superior performance is not only due to the parameter size but also to the effective use of the tensorized attention mechanism. To evaluate the scalability of TAM, we also conducted experiments with the number of layers reduced to 8, as shown in Figure 16. The results showed that the accuracy of TAM improved when the number of layers increased from 8 to 12, indicating that increasing the number of parameters has a positive impact on the performance of TAM.

[0169] In the area of ​​response selection models, BERT-FP ("Fine-grained post-training for improving retrieval-based dialogue systems," Janghoon Han et al., 2021) currently stands out across a variety of datasets. Because BERT-FP is built on BERT, the process of integrating TAM into BERT-FP is intuitive. While BERT requires slightly fewer parameters and slightly faster computation time in short context-response scenarios, scenarios such as the NFL dataset (Pretraining per epoch: BERT 19 min, TAM 33 min; Finetuning per epoch: BERT 24 min, TAM 27 min; Memory: BERT 20,781 MiB, TAM 25,381 MiB) consistently demonstrate superior accuracy for TAM within realistic timeframes. This advantage is evident from the results shown in Figures 10 and 13. Furthermore, TAM is faster than BERT, especially in longer context-response scenarios. This is demonstrated on the Politics dataset (Pretraining per epoch: BERT 33 min, TAM 29 min; Finetuning per epoch: BERT 49 min, TAM 46 min; Memory: BERT 51,531 MiB, TAM 62,817 MiB). Furthermore, when exploring Transformer Encoder models for response selection other than BERT, there are alternatives such as RoBERTa. However, when applying TAM to RoBERTa, the impact on computation time and memory efficiency is expected to be similar to that of BERT, since these models share the same underlying Transformer architecture.

[0170] As mentioned above, to demonstrate the effectiveness of the tensorized attention model, we integrated it into a Transformer encoder and evaluated its performance in response selection and question answering tasks. Following the approach of [2], our experiments primarily focused on measuring the impact of multidimensional attention in the tensorized attention model within the encoder model. The above experiments were conducted in two passes: training the tensorized attention model from scratch and combining it with a pre-trained Transformer model. Experiments on the NFL and Politics datasets and the TweetQA dataset showed that the tensorized attention model consistently outperformed existing Transformer-based methods in terms of accuracy.

[0171] [Effects] As described above, in the information processing device 100 of this embodiment, the acquisition unit 131 acquires a model that receives first information corresponding to a first object and second information corresponding to a second object and added to the first information as input, and outputs third information indicating an inference result for the first information, the model including an attention mechanism that accepts a first input corresponding to the first information and a second input corresponding to the second information. Furthermore, the inference unit 132 inputs, to the model, target first information that is the first information to be inferred and target second information that is second information corresponding to the target first information, and causes the model to output target third information that is an inference result for the target first information, thereby generating target third information that reflects the target second information by the attention mechanism.

[0172] Specifically, the acquisition unit 131 acquires a model M1, which is a model with a semantic attention mechanism (tensorized attention model). The inference unit 132 uses the model M1 to infer an answer (third information) to a question (first information) based on the question and semantic information (second information) corresponding to the question, and generates an answer that reflects the semantic information by the attention mechanism.

[0173] In this way, when inferring third information for the first information, the information processing device 100 uses a model (tensorized attention model) including an attention mechanism that accepts second input corresponding to second information, which is information different from the first information and the third information, to perform inference processing that also takes into account the second information, thereby enabling inference that appropriately reflects information other than the information input as the inference target. In this way, the tensorized attention model extends the existing Tensorized Transformer architecture to include a Tensorized Attention Model for efficiently integrating attention weights in multi-object relationships. The tensorized attention model utilizes Tucker decomposition to calculate attention weights between various object types and seamlessly integrate them into the Transformer encoder.

[0174] [Program] A program written in a computer-executable language may be created to execute the processes performed by the information processing device 100 according to the above embodiment. In one embodiment, the information processing device 100 can be implemented by installing an information processing program that executes the above information processing as package software or online software on a desired computer. For example, by executing the above information processing program on an information processing device, the information processing device can function as the information processing device 100. Other examples of information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants). The functions of the information processing device 100 may also be implemented on a cloud server.

[0175] 7 is a diagram showing an example of a computer that executes an information processing program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0176] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.

[0177] Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The various pieces of information described in the above embodiments are stored in the hard disk drive 1031 or memory 1010, for example.

[0178] The information processing program is stored in the hard disk drive 1031 as, for example, a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the information processing device 100 described in the above embodiment is written is stored in the hard disk drive 1031.

[0179] Furthermore, data used for information processing by the information processing program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0180] The program module 1093 and program data 1094 related to the information processing program are not limited to being stored in the hard disk drive 1031, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1041. Alternatively, the program module 1093 and program data 1094 related to the information processing program may be stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0181] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0182] REFERENCE SIGNS LIST 100 Information processing device 110 Communication unit 120 Storage unit 121 Data storage unit 122 Model information storage unit 130 Control unit 131 Acquisition unit 132 Inference unit 133 Learning unit 134 Provision unit

Claims

1. An information processing device comprising: an acquisition unit that acquires a model that receives first information corresponding to a first object and second information corresponding to a second object and added to the first information as input, and outputs third information indicating an inference result for the first information, the model including an attention mechanism that accepts a first input corresponding to the first information and a second input corresponding to the second information; and an inference unit that inputs target first information, which is the first information to be inferred, and target second information, which is the second information corresponding to the target first information, to the model acquired by the acquisition unit, and causes the model to output target third information, which is an inference result for the target first information, thereby generating the target third information in which the target second information is reflected by the attention mechanism.

2. The information processing device described in claim 1, characterized in that the acquisition unit acquires the model including the attention mechanism capable of processing data of different dimensional sizes using Tucker decomposition, and the inference unit inputs the target first information and the target second information into the model, and generates the target third information by having the model output the target third information as the data of different dimensional sizes is processed in the attention mechanism.

3. An information processing method comprising: an acquisition step of acquiring a model that receives first information corresponding to a first object and second information corresponding to a second object and added to the first information as input, and outputs third information indicating an inference result for the first information, the model including an attention mechanism that accepts a first input corresponding to the first information and a second input corresponding to the second information; and an inference step of inputting target first information, which is the first information to be inferred, and target second information, which is the second information corresponding to the target first information, into the model acquired by the acquisition step, and having the model output target third information, which is an inference result for the target first information, thereby generating the target third information in which the target second information is reflected by the attention mechanism.

4. An information processing program that causes a computer to execute the following steps: an acquisition procedure for acquiring a model that receives first information corresponding to a first object and second information corresponding to a second object and added to the first information as input, and outputs third information indicating an inference result for the first information, the model including an attention mechanism that accepts a first input corresponding to the first information and a second input corresponding to the second information; and an inference procedure for inputting target first information, which is the first information to be inferred, and target second information, which is the second information corresponding to the target first information, into the model acquired by the acquisition procedure, and having the model output target third information, which is an inference result for the target first information, thereby generating the target third information in which the target second information is reflected by the attention mechanism.

Images