Information processing device, information processing method, and information processing program

Abstract

A causal relationship estimation unit (111) estimates, with sequence data indicating sequences of a plurality of elements as input, causal relationships between the plurality of elements by using a causal relationship estimation model. A chunk determination unit (112) determines, for each of the elements, a chunk serving as a calculation range of a self-attention mechanism on the basis of the estimated causal relationships. A self-attention calculation unit (113) calculates, with the chunk of each of the plurality of elements as the calculation range, a value of a self-attention of each of the plurality of elements, by means of the self-attention mechanism. A feature amount calculation unit (114) calculates, with the value of the self-attention of each of the plurality of elements as input, a feature amount of each of the plurality of elements by using an inference model.

Description

Information processing device, information processing method, and information processing program

[0001] This disclosure relates to causal reasoning.

[0002] In conventional models using attention mechanisms, the challenge has been to balance the increase in computational complexity with the improvement of prediction accuracy. In a typical attention mechanism, weighting is performed on each element of the input sequence data, resulting in a computational complexity of O(n ２ ) increases quadratically, where "n" represents the length of the sequence data. This increase in computational complexity leads to problems such as increased model inference time in large-scale data or real-time applications. Furthermore, for attention mechanisms to function properly in improving prediction accuracy, it is necessary to accurately grasp the structural causal relationships hidden in the sequence data and determine the appropriate computation scope. However, conventional methods often fail to adequately grasp causal relationships and determine the computation scope, limiting the improvement of prediction accuracy.

[0003] Patent Literature 1 discloses a model acquisition method that improves model performance while reducing the consumption of computational resources, etc. This method trains a model according to a learning goal corresponding to syntactic information for a self-attention module.

[0004] Patent No. 7379792

[0005] An object of the present disclosure is to enable calculation of feature quantities for each of multiple elements shown in sequence data while suppressing an increase in the amount of calculation.

[0006] The information processing device of the present disclosure includes: a causal relationship estimation unit that receives sequence data indicating a sequence of multiple elements as input and estimates the causal relationship between the multiple elements using a causal relationship estimation model; a chunk determination unit that determines a chunk that will be the calculation range of a self-attention mechanism for each of the elements based on the estimated causal relationship; a self-attention calculation unit that uses the self-attention mechanism to calculate a self-attention value for each of the multiple elements with the chunk of each of the multiple elements as the calculation range; and a feature calculation unit that receives the self-attention value for each of the multiple elements as input and calculates a feature of each of the multiple elements using an inference model.

[0007] According to the present disclosure, chunks are determined based on the causal relationships between multiple elements in sequential data, which makes it possible to calculate the feature quantities of each of the multiple elements in the sequential data while minimizing the increase in the amount of calculations.

[0008] FIG. 1 is a configuration diagram of an information processing device 100 according to the first embodiment. FIG. 2 is a configuration diagram of an inference unit 110 according to the first embodiment. FIG. 3 is a configuration diagram of a learning unit 120 according to the first embodiment. FIG. 4 is a flowchart of inference processing according to the first embodiment. FIG. 5 is a diagram showing an example of a causal relationship graph 192 and a matrix 193 according to the first embodiment. FIG. 6 is a diagram showing an example of a matrix 193 according to the first embodiment. FIG. 7 is a diagram showing an example of a parent-child node group 195 and a chunk 194 according to the first embodiment. FIG. 8 is a diagram showing an example of a chunk 194 according to the first embodiment. FIG. 9 is a diagram showing an example of a chunk 194 according to the first embodiment. FIG. 10 is a diagram showing a flowchart of learning processing according to the first embodiment. FIG. 11 is a diagram showing a flowchart of learning processing according to the first embodiment. FIG. 12 is a hardware configuration diagram of an information processing device 100 according to the first embodiment.

[0009] In the embodiments and drawings, the same or corresponding elements are denoted by the same reference numerals. The description of elements denoted by the same reference numerals as those already described will be omitted or simplified as appropriate. Arrows in the drawings primarily indicate the flow of data or the flow of processing.

[0010] First Embodiment A first embodiment of calculating the feature quantities of each of a plurality of elements shown in sequence data while suppressing an increase in the amount of calculation will be described with reference to FIGS.

[0011] ***Description of Configuration*** The configuration of the information processing device 100 will be described with reference to Fig. 1. The information processing device 100 is a computer equipped with hardware such as a processor 101, a memory 102, an auxiliary storage device 103, a communication device 104, and an input / output interface 105. These pieces of hardware are connected to one another via signal lines.

[0012] The processor 101 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 101 is a CPU, a DSP, or a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0013] The memory 102 is a volatile or non-volatile storage device. The memory 102 is also called a primary storage device or a main memory. For example, the memory 102 is a RAM. Data stored in the memory 102 is saved in the secondary storage device 103 as needed. RAM is an abbreviation for Random Access Memory.

[0014] The auxiliary storage device 103 is a non-volatile storage device. For example, the auxiliary storage device 103 is a ROM, a HDD, a flash memory, or a combination of these. Data stored in the auxiliary storage device 103 is loaded into the memory 102 as needed. ROM is an abbreviation for Read Only Memory. HDD is an abbreviation for Hard Disk Drive.

[0015] The communication device 104 is a receiver and a transmitter. For example, the communication device 104 is a communication chip or a NIC. Communication of the information processing device 100 is performed using the communication device 104. NIC is an abbreviation for Network Interface Card.

[0016] The input / output interface 105 is a port to which an input device and an output device are connected. For example, the input / output interface 105 is a USB terminal, the input devices are a keyboard and a mouse, and the output device is a display. Input and output of the information processing device 100 is performed using the input / output interface 105. USB is an abbreviation for Universal Serial Bus.

[0017] The information processing device 100 includes elements such as an inference unit 110 and a learning unit 120. These elements are realized by software.

[0018] The auxiliary storage device 103 stores an information processing program for causing the computer to function as the inference unit 110 and the learning unit 120. The information processing program is loaded into the memory 102 and executed by the processor 101. The auxiliary storage device 103 also stores an OS. At least a portion of the OS is loaded into the memory 102 and executed by the processor 101. The processor 101 executes the information processing program while running the OS. OS is an abbreviation for Operating System.

[0019] Data (input data, output data, etc.) of the information processing program is stored in the storage unit 190. The memory 102 functions as the storage unit 190. However, a storage device such as the auxiliary storage device 103, a register in the processor 101, or a cache memory in the processor 101 may function as the storage unit 190 instead of or together with the memory 102.

[0020] The information processing program can be recorded (stored) in a computer-readable manner on a non-volatile recording medium such as an optical disk or a flash memory.

[0021] The configuration of the inference unit 110 will be described with reference to Fig. 2. The inference unit 110 includes a causal relationship estimation unit 111, a chunk determination unit 112, a self-attention calculation unit 113, and a feature calculation unit 114. The parameter data 191, the sequence data 181, and the inference data 182 are stored in the storage unit 190.

[0022] The configuration of the learning unit 120 will be described with reference to Fig. 3. The learning unit 120 includes a causal relationship estimation unit 121, a chunk determination unit 122, a self-attention calculation unit 123, a feature calculation unit 124, and a learning control unit 125. Learning data 183 and correct answer data 184 are stored in a storage unit 190.

[0023] ***Description of Operation*** The operational procedure of the information processing device 100 corresponds to an information processing method. Also, the operational procedure of the information processing device 100 corresponds to a processing procedure according to an information processing program.

[0024] The inference process of the information processing method will be described with reference to Fig. 4. Trained parameters of the causal relationship estimation model and the inference model are stored in parameter data 191. For example, parameters of a neural network (NN) used in each model are stored in the parameter data 191. The causal relationship estimation model and the inference model will be described later.

[0025] In step S111, the causal relationship estimation unit 111 acquires parameters of the causal relationship estimation model from the parameter data 191 and sets them in the causal relationship estimation model. In addition, the feature calculation unit 114 acquires parameters of the inference model from the parameter data 191 and sets them in the inference model.

[0026] In step S112, the causal relationship estimation unit 111 acquires the sequence data 181. For example, a user inputs the sequence data 181 into the information processing device 100, and the causal relationship estimation unit 111 receives the input sequence data 181.

[0027] Sequence data is data that indicates a sequence of multiple elements. For example, sequence data indicates multiple numbers arranged in chronological order or multiple words arranged in a sentence. Examples of sequence data include natural language text data or FA sensor data. FA is an abbreviation for factory automation.

[0028] The causal relationship estimation unit 111 then receives the sequence data 181 and estimates the causal relationships among multiple elements using the causal relationship model, thereby creating a causal relationship graph 192 as data indicating the estimated causal relationships.

[0029] The causal relationship model will now be described. The causal relationship model is a machine-learned model that estimates the causal relationships between multiple elements indicated in input data. The causal relationship model is stored in the storage unit 190. The causal relationship model estimates the causal relationships using, for example, a convex optimization method. The causal relationship model outputs a causal relationship graph 192 as data indicating the estimated causal relationships.

[0030] The causal relationship graph 192 shows the linear structural causal relationships between multiple elements. Specifically, the causal relationship graph 192 shows multiple elements as nodes, with nodes having a direct causal relationship connected by paths.

[0031] The causality graph 192 includes a path coefficient for each path, which indicates the magnitude of the causal relationship between the nodes connected by the path.

[0032] The causal graph 192 can be represented by a matrix 193 .

[0033] 5 shows an example of a causal relationship graph 192 and a matrix 193. The sequence data 181 indicates a sequence of nine elements. The nine elements are extracted using, for example, a neural network.

[0034] The causal relationship graph 192 has nine nodes (boxes) representing nine elements. In the causal relationship graph 192, nodes having a direct causal relationship are connected by paths (arrows). Each path is assigned a path coefficient indicating the magnitude of the causal relationship between the nodes connected by the path. Of two nodes connected by a path, one node (the starting point of the arrow) is called the parent node, and the other node (the ending point of the arrow) is called the child node. For example, the sixth element node has a direct causal relationship with each of the third element node, the fifth element node, the eighth element node, and the ninth element node. The third element node is the parent node of the sixth element. The fifth element node, the eighth element node, and the ninth element node are child nodes of the sixth element.

[0035] The matrix 193 has nine elements and nine queries (rows) and nine elements and nine keys (columns). Attention, which will be described later, is calculated using the keys that are parent nodes or child nodes of the queries. The shading in the matrix 193 indicates pairs of nodes that have a direct causal relationship.

[0036] 6 shows an example of matrix 193. Matrix 193 can be replaced with matrix 193I and matrix 193T. Matrix 193I is obtained by adding an identity matrix to matrix 193. Matrix 193T is obtained by adding a transposed matrix and an identity matrix to matrix 193.

[0037] 4, the description will continue from step S113. In step S113, the chunk determination unit 112 determines a chunk 194 for each element based on the estimated causal relationship.

[0038] Chunk 194 is the computational domain of the self-attention mechanism.

[0039] Specifically, the chunk determination unit 112 selects a parent-child node group 195 from the causal relationship graph 192 for each element, and determines a chunk 194 for each element based on the parent-child node group 195. The parent-child node group 195 consists of a target element node representing the target element and nodes in a parent-child relationship with the target element node.

[0040] For example, the chunk determination unit 112 determines the chunk 194 based on the parent-child node group 195 as follows: The chunk determination unit 112 determines an element group corresponding to the parent-child node group 195 as the chunk 194. FIG. 7 shows an example of the parent-child node group 195 and the chunk 194. The parent-child node group 195 of the sixth element is the third element node, the fifth element node, the sixth element node, the eighth element node, and the ninth element node. In this case, the third element, the fifth element, the sixth element, the eighth element, and the ninth element corresponding to the parent-child node group of the sixth element become the chunk 194 of the sixth element. The length of the chunk 194 of the sixth element is 5.

[0041] Returning to FIG. 4 , the description of step S113 will be continued. For example, the chunk determination unit 112 determines the chunk 194 based on the parent-child node group 195 as follows. The chunk determination unit 112 compares the number of nodes in the parent-child node group 195 with a constant. The constant is a predetermined fixed value. If the number of nodes in the parent-child node group 195 is the same as the constant, the chunk determination unit 112 determines the element group corresponding to the parent-child node group 195 as the chunk 194. If the number of nodes in the parent-child node group 195 is less than the constant, the chunk determination unit 112 determines the element group corresponding to the parent-child node group 195 and one or more dummy elements as the chunk 194. If the number of nodes in the parent-child node group 195 exceeds the constant, the chunk determination unit 112 determines the same number of elements as the constant from the element group corresponding to the parent-child node group 195 as the chunk 194. Specifically, the chunk determination unit 112 selects, as chunks 194 , the same number of elements as a constant from the element group corresponding to the parent-child node group 195 based on the path coefficient group for the parent-child node group 195 .

[0042] FIG. 8 shows an example of a chunk 194. The parent-child node group 195 of the first element is made up of the first element node, the seventh element node, the fourth element node, and the second element node. The number of nodes in the parent-child node group 195 of the first element is four. When the constant is four, the number of nodes in the parent-child node group 195 of the first element is the same as the constant. In this case, the first element, the seventh element, the fourth element, and the second element become the chunk 194 of the first element. Note that in the chunk 194, the remaining elements (the seventh element, the fourth element, and the second element) excluding the target element (the first element) are sorted in descending order of path coefficient.

[0043] FIG. 9 shows an example of chunk 194. The parent-child node group 195 of the second element includes the second element node, the third element node, and the first element node. The number of nodes in the parent-child node group 195 of the second element is three. If the constant is four, the number of nodes in the parent-child node group 195 of the second element is less than the constant. In this case, the second element, the third element, the first element, and the zeroth element become chunk 194 of the second element. The zeroth element is a dummy element that is complemented by chunk 194. Note that in chunk 194, the remaining elements (first element, third element) excluding the target element (second element) and the dummy element (zeroth element) are sorted in descending order of path coefficient.

[0044] FIG. 10 shows an example of a chunk 194. The parent-child node group 195 of the sixth element consists of the sixth element node, the ninth element node, the eighth element node, the third element node, and the fifth element node. The number of nodes in the parent-child node group 195 of the sixth element is five. If the constant is four, the number of nodes in the parent-child node group 195 of the sixth element exceeds the constant. In this case, the parent-child node group 195 of the sixth element is sorted in descending order of path coefficient. Then, elements equal in number to the constant are selected in descending order of path coefficient, and the selected elements become the chunk 194 of the sixth element. The chunk 194 of the sixth element consists of the sixth element, the ninth element, the eighth element, and the third element. The fifth element is not included in the chunk 194 of the sixth element. In other words, the fifth element is not used.

[0045] 4, the description will continue from step S114. In step S114, the self-attention calculation unit 113 calculates the self-attention value of each of the plurality of elements by the self-attention mechanism, with each chunk 194 of the plurality of elements as the calculation range.

[0046] The self-attention mechanism is realized by an algorithm for calculating the self-attention value.

[0047] In step S115, the feature calculation unit 114 calculates the feature of each of the plurality of elements using the inference model with the self-attention values ​​of each of the plurality of elements as input, thereby generating inference data 182 as data indicating the feature of each of the plurality of elements.

[0048] The inference model will now be described. The inference model is a machine-learned model that calculates multiple feature quantities corresponding to multiple self-attention values ​​indicated in input data and calculates an inference result based on the multiple feature quantities. The feature quantities are also called latent features. The inference result varies depending on the task and may indicate a prediction result or a classification result. The inference model outputs inference data 182 as data indicating the multiple feature quantities and the inference result.

[0049] The learning process of the information processing method will be described with reference to Figures 11 and 12. In step S121, the causal relationship estimation unit 121 initializes parameters of the causal relationship estimation model. In addition, the feature calculation unit 114 initializes parameters of the inference model.

[0050] In step S122, the causal relationship estimation unit 121 acquires the training data 183. For example, a user inputs the training data 183 to the information processing device 100, and the causal relationship estimation unit 111 receives the input training data 183.

[0051] The learning data 183 is sequence data for learning.

[0052] The causal relationship estimation unit 111 then estimates the causal relationships of the multiple elements for learning using a causal relationship model, with the input being a sequence of multiple elements for learning indicated in the learning data 183. As a result, a causal relationship graph 192 is created as data indicating the estimated causal relationships.

[0053] In step S123, the chunk determination unit 122 determines a chunk 194 for each element based on the estimated causal relationship. The determination method is the same as the method in step S113.

[0054] In step S124, the self-attention calculation unit 123 calculates the self-attention value of each of the plurality of elements for learning by using the self-attention mechanism, with each chunk 194 of the plurality of elements for learning as the calculation range.

[0055] In step S125, the feature calculation unit 124 calculates the feature of each of the multiple elements for training using the inference model with the self-attention values ​​of each of the multiple elements for training as input, thereby generating inference data 182.

[0056] In step S126, the learning control unit 125 determines whether the learning has converged based on the feature amounts of each of the multiple elements calculated in step S125.

[0057] Convergence of learning is determined as follows: First, the learning control unit 125 acquires the correct answer data 184. For example, a user inputs the correct answer data 184 to the information processing device 100, and the learning control unit 125 receives the input correct answer data 184.

[0058] The correct answer data 184 indicates the correct answers of a plurality of feature quantities and the correct answers of the inference results.

[0059] Next, the learning control unit 125 compares the inference data 182 with the supervised data 184. Then, based on the comparison result, the learning control unit 125 determines whether the learning has converged. For example, the learning control unit 125 calculates a feature difference. The feature difference indicates the magnitude of the difference between the multiple feature amounts indicated in the inference data 182 and the multiple feature amounts indicated in the supervised data 184. Then, the learning control unit 125 compares the feature difference with a threshold, and determines that the learning has converged if the feature difference is smaller than the threshold.

[0060] If the learning has converged, the process proceeds to step S127. If the learning has not converged, the learning control unit 125 updates the parameters of the causal relationship estimation model and the inference model based on the comparison result. Thereafter, the process proceeds to step S122.

[0061] In step S127, the causal relationship estimation unit 121 generates sequence data representing a sequence of multiple feature quantities corresponding to the multiple elements for training as a sequence of multiple new elements for training. The causal relationship estimation unit 121 then inputs the generated sequence data and estimates the causal relationships among the multiple new elements for training using a causal relationship estimation model.

[0062] In step S128, the learning control unit 125 determines whether there is a large difference between the causal relationship estimated in step S127 and the causal relationship estimated in step S122.

[0063] The magnitude of the difference between the causal relationships is determined as follows. First, the learning control unit 125 updates the causal relationship graph 192 created in step S122 to the causal relationship graph 192 created in step S127, and calculates the graph update amount. The graph update amount indicates the amount of update from the causal relationship graph 192 created in step S122 to the causal relationship graph 192 created in step S127. The learning control unit 125 then compares the graph update amount with a threshold, and determines that the difference between the causal relationships is large if the graph update amount is greater than the threshold.

[0064] If the difference between the causal relationships is large, the learning control unit 125 updates the parameters of the causal relationship estimation model and the inference model based on the comparison result. Then, the process proceeds to step S122. If the difference between the causal relationships is not large, the process proceeds to step S129.

[0065] In step S129, the causal relationship estimation unit 111 stores the parameters of the causal relationship estimation model in the parameter data 191. In addition, the feature amount calculation unit 114 stores the parameters of the inference model in the parameter data 191.

[0066] The learning process does not have to include steps S127 and S128. In this case, if the learning has converged in step S126, the process proceeds to step S129.

[0067] ***Effects of Embodiment 1*** In embodiment 1, when learning and inferring a predictive model using an attention mechanism, the range of feature calculation is automatically determined based on causal relationships, thereby improving the accuracy of the predictive model.

[0068] In this paper, we propose a new method to simultaneously reduce the computational complexity of the attention mechanism and improve prediction accuracy. A system that combines causal relationship estimation, self-attention chunk determination, and self-attention value calculation enables appropriate control of the computation range, which is expected to improve the performance of the prediction model.

[0069] In a model using an attention mechanism, accuracy may decrease if feature correlation is insufficient. In the first embodiment, a causal graph is estimated from input data, and the learned parameters are appropriately saved and used to improve the performance of the model during learning and inference.

[0070] In the first embodiment, the causal relationship estimation means and the latent feature calculation means are appropriately trained during training, and during inference, these trained parameters are read from the DB to estimate causal relationships and calculate feature quantities, thereby improving the performance of the model.

[0071] According to the first embodiment, it is possible to automatically determine the feature calculation range based on the causal relationships during model training and inference, thereby improving the prediction accuracy of the model. Furthermore, linear structural learning of the causal relationship estimation means and the latent feature calculation means allows the feature calculation of the model to be performed appropriately, improving the performance of the prediction model.

[0072] Summary of First Embodiment The information processing device 100 is a system that includes a causal relationship estimation unit 111, a chunk determination unit 112, a self-attention calculation unit 113, and a feature calculation unit 114. These components work together to automatically and effectively calculate features. The causal relationship estimation unit 111 estimates a causal graph from input data. The chunk determination unit 112 identifies a range in which direct causal relationships in the data can be elucidated based on the obtained causal graph. The self-attention calculation unit 113 calculates a self-attention value from that range. The feature calculation unit 114 extracts latent features through machine learning. A system that includes these components automatically and effectively obtains useful information, including self-attention, from input data, and builds an advanced prediction model.

[0073] A convex optimization method is used to determine the range of self-attention chunks, and the causal relationships of the input data are determined. When determining the range of chunks, data with direct causal relationships are selected, and the range of chunks is determined based on that data. A constant is set as the length of the chunks, and if the length of the chunks is less than the constant, they are sorted in descending order of the path coefficients of the causal graph, and missing nodes are supplemented with zero nodes. If the length of the chunks exceeds the constant, they are sorted in descending order of the path coefficients of the causal graph, and the excess nodes are not included in the chunks. The convex optimization method is combined with a chunk determination method that utilizes causal relationship data to appropriately control the calculation range and achieve efficient and effective self-attention calculations.

[0074] *** Supplementary Note on First Embodiment *** The first embodiment achieves the following features, and is expected to improve the performance of the prediction model. (1) Consideration of correlation and determination of direct causal relationships In order to consider the correlation of the features used in prediction, direct causal relationships between nodes in the input sequence data are determined. To calculate the self-attention feature, the calculation range (chunk) is automatically determined from input data (key, query) of length n. (2) Improvement of prediction accuracy The structural causal relationships of the input sequence data (key, query) are estimated, and the range of chunks is appropriately determined based on the parent-child relationships of the causal relationships. This allows the model to capture important correlations, and high prediction accuracy is expected. (3) Reduction of the amount of calculation during inference By appropriately determining the calculation range (chunk) based on the estimation of causal relationships, the amount of calculation during inference is significantly reduced. Specifically, the amount of calculation is reduced to O(n ２ ) to O(n k), where "k" indicates the chunk size. This enables efficient inference, improves real-time performance, and enables adaptation to large-scale data. These features allow a prediction model to be constructed that properly captures the characteristics of the input sequence data and takes into account the correlation between nodes, improving prediction accuracy.

[0075] The hardware configuration of the information processing device 100 will be described with reference to Fig. 13. The information processing device 100 includes a processing circuit 109. The processing circuit 109 is hardware that realizes an inference unit 110 and a learning unit 120. The processing circuit 109 may be dedicated hardware, or may be a processor 101 that executes a program stored in a memory 102.

[0076] When the processing circuit 109 is dedicated hardware, the processing circuit 109 may be, for example, a single circuit, a multiple circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array.

[0077] The information processing device 100 may include a plurality of processing circuits that replace the processing circuit 109 .

[0078] In the processing circuit 109, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

[0079] In this way, the functions of the information processing device 100 can be realized by hardware, software, firmware, or a combination of these.

[0080] The first embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the present disclosure. The first embodiment may be implemented in part or in combination with other embodiments. The procedures described using flowcharts and the like may be modified as appropriate.

[0081] The "unit" of each element of the information processing device 100 may be read as a "process," a "step," a "circuit," or a "circuitry."

[0082] Aspects of the present disclosure are described below as appendices. (Appendix 1) An information processing device comprising: a causal relationship estimation unit that receives sequence data indicating a sequence of a plurality of elements as an input and estimates a causal relationship between the plurality of elements using a causal relationship estimation model; a chunk determination unit that determines, for each of the elements, a chunk that will be a calculation range of a self-attention mechanism based on the estimated causal relationship; a self-attention calculation unit that calculates a self-attention value for each of the plurality of elements using the self-attention mechanism, with the chunk of each of the plurality of elements as the calculation range; and a feature calculation unit that receives the self-attention value for each of the plurality of elements as an input and calculates a feature value for each of the plurality of elements using an inference model.

[0083] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the causal relationship estimation model is a model that estimates causal relationships using a convex optimization method.

[0084] (Supplementary Note 3) The information processing device described in Supplementary Note 1 or Supplementary Note 2, wherein the causal relationship estimation unit creates a causal relationship graph as data indicating the estimated causal relationship, in which each of the plurality of elements is a node and nodes having a direct causal relationship are connected by paths, and the chunk determination unit selects, for each element, from the causal relationship graph, a node representing the element and a node having a parent-child relationship with the node of the element as a parent-child node group, and determines the chunk for each element based on the parent-child node group.

[0085] (Supplementary Note 4) The information processing device according to Supplementary Note 3, wherein the chunk determination unit determines an element group corresponding to the parent-child node group as the chunk.

[0086] (Supplementary Note 5) The information processing device described in Supplementary Note 3, wherein the chunk determination unit determines a group of elements corresponding to the parent-child node group as the chunk when the number of nodes in the parent-child node group is the same as a constant, determines the group of elements corresponding to the parent-child node group and one or more dummy elements as the chunk when the number of nodes in the parent-child node group is less than the constant, and determines elements of the group of elements corresponding to the parent-child node group that are the same number as the constant as the chunk when the number of nodes in the parent-child node group exceeds the constant.

[0087] (Supplementary Note 6) The information processing device described in Supplementary Note 5, wherein the causal graph includes, for each path, a path coefficient indicating the magnitude of the causal relationship between the nodes connected by the path, and when the number of nodes in the parent-child node group exceeds the constant, the chunk determination unit selects, as the chunk, from the element group corresponding to the parent-child node group, a number of elements equal to the constant based on a path coefficient group for the parent-child node group.

[0088] (Supplementary Note 7) An information processing method comprising: inputting sequence data indicating a sequence of a plurality of elements and using a causal relationship estimation model to estimate a causal relationship between the plurality of elements; determining a chunk that will be the calculation range of a self-attention mechanism for each of the elements based on the estimated causal relationship; calculating a self-attention value for each of the plurality of elements using the self-attention mechanism with the chunk of each of the plurality of elements as the calculation range; and inputting the self-attention value for each of the plurality of elements and using an inference model to calculate a feature quantity for each of the plurality of elements.

[0089] (Supplementary Note 8) An information processing program for causing a computer to execute the following steps: a causal relationship estimation process that takes as input sequence data indicating a sequence of multiple elements and estimates the causal relationship of the multiple elements using a causal relationship estimation model; a chunk determination process that determines, for each of the elements, a chunk that will be the calculation range of a self-attention mechanism based on the estimated causal relationship; a self-attention calculation process that takes the chunk of each of the multiple elements as the calculation range and calculates a self-attention value of each of the multiple elements using the self-attention mechanism; and a feature calculation process that takes as input the self-attention value of each of the multiple elements and calculates a feature of each of the multiple elements using an inference model.

[0090] 100 Information processing device, 101 Processor, 102 Memory, 103 Auxiliary storage device, 104 Communication device, 105 Input / output interface, 109 Processing circuit, 110 Inference unit, 111 Causal relationship estimation unit, 112 Chunk determination unit, 113 Self-attention calculation unit, 114 Feature calculation unit, 120 Learning unit, 121 Causal relationship estimation unit, 122 Chunk determination unit, 123 Self-attention calculation unit, 124 Feature calculation unit, 181 Sequence data, 182 Inference data, 183 Learning data, 184 Correct answer data, 190 Memory unit, 191 Parameter data, 192 Causal relationship graph, 193 Matrix, 194 Chunk, 195 Parent-child node group.

Claims

1. An information processing device comprising: a causal relationship estimation unit that receives as input sequence data indicating a sequence of multiple elements and estimates the causal relationships of the multiple elements using a causal relationship estimation model; a chunk determination unit that determines, for each of the elements, a chunk that will be the calculation range of a self-attention mechanism based on the estimated causal relationships; a self-attention calculation unit that uses the self-attention mechanism to calculate a self-attention value for each of the multiple elements with the chunk of each of the multiple elements as the calculation range; and a feature calculation unit that receives as input the self-attention value for each of the multiple elements and calculates a feature value for each of the multiple elements using an inference model.

2. The information processing device according to claim 1, wherein the causal relationship estimation model is a model that estimates causal relationships using a convex optimal method.

3. The information processing device described in claim 1 or claim 2, wherein the causal relationship estimation unit creates a causal relationship graph as data indicating the estimated causal relationship, with each of the multiple elements as a node and nodes having a direct causal relationship connected by paths, and the chunk determination unit selects, for each element, from the causal relationship graph a node representing the element and a node in a parent-child relationship with the node of the element as a parent-child node group, and determines the chunk for each element based on the parent-child node group.

4. The information processing device according to claim 3, wherein the chunk determination unit determines an element group corresponding to the parent-child node group as the chunk.

5. The information processing device according to claim 3, wherein the chunk determination unit determines the group of elements corresponding to the parent-child node group as the chunk when the number of nodes in the parent-child node group is the same as a constant, determines the group of elements corresponding to the parent-child node group and one or more dummy elements as the chunk when the number of nodes in the parent-child node group is less than the constant, and determines the same number of elements as the constant from the group of elements corresponding to the parent-child node group as the chunk when the number of nodes in the parent-child node group exceeds the constant.

6. The information processing device described in claim 5, wherein the causal graph includes, for each path, a path coefficient indicating the magnitude of the causal relationship between the nodes connected by the path, and when the number of nodes in the parent-child node group exceeds the constant, the chunk determination unit selects as the chunk a number of elements equal to the constant from the element group corresponding to the parent-child node group based on the path coefficient group for the parent-child node group.

7. An information processing method that takes as input sequence data indicating a sequence of multiple elements and uses a causal relationship estimation model to estimate the causal relationships of the multiple elements, determines a chunk that will be the calculation range of a self-attention mechanism for each of the elements based on the estimated causal relationships, calculates a self-attention value for each of the multiple elements using the self-attention mechanism with the chunk of each of the multiple elements as the calculation range, and inputs the self-attention value for each of the multiple elements and uses an inference model to calculate the feature values ​​of each of the multiple elements.

8. An information processing program for causing a computer to execute the following steps: a causal relationship estimation process that takes as input sequence data indicating a sequence of multiple elements and estimates the causal relationship between the multiple elements using a causal relationship estimation model; a chunk determination process that determines, for each of the elements, a chunk that will be the calculation range of a self-attention mechanism based on the estimated causal relationship; a self-attention calculation process that uses the chunk of each of the multiple elements as the calculation range to calculate a self-attention value for each of the multiple elements using the self-attention mechanism; and a feature calculation process that takes as input the self-attention value for each of the multiple elements and calculates a feature value for each of the multiple elements using an inference model.

Images