Information processing device, information processing method and information processing program

By combining causal reasoning and self-attention mechanisms, the problem of balancing computational cost and prediction accuracy in attention mechanisms is solved, achieving efficient feature computation and improved prediction model performance.

CN123125591APending Publication Date: 2026-11-17MITSUBISHI ELECTRIC CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202480091106.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-04-25
Filing Date
2024-07-05
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In models using attention mechanisms, it is difficult to balance increased computational cost with improved prediction accuracy, especially in large-scale data or real-time applications where inference time increases and insufficient understanding of causal relationships and determination of computational scope leads to limited prediction accuracy.

Method used

The causal relationship derivation unit derives the causal relationship of the sequence data, the block decision unit determines the computational range of the self-attention mechanism, the self-attention computation unit calculates the feature quantity, the feature quantity computation unit calculates the feature quantity, and the convex optimization method is used to determine the range of the block in order to control the computational load.

Benefits of technology

It effectively suppressed the increase in computational load, improved the accuracy of the prediction model, adapted to large-scale data and improved real-time performance, and achieved efficient feature quantity calculation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125591A_ABST
    Figure CN123125591A_ABST
Patent Text Reader

Abstract

The causal relationship derivation unit (111) takes sequence data representing a sequence of multiple elements as input and uses a causal relationship derivation model to deduce the causal relationship between the multiple elements. The block determination unit (112) determines, based on the derived causal relationship, the block that becomes the computational scope of the self-attention mechanism for each element. The self-attention calculation unit (113) takes the respective blocks of the multiple elements as the computational scope and calculates the self-attention value of each of the multiple elements using the self-attention mechanism. The feature quantity calculation unit (114) takes the self-attention value of each of the multiple elements as input and uses an inference model to calculate the feature quantity of each of the multiple elements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to inferences based on causality. Background Technology

[0002] In existing models that use attention mechanisms, achieving a balance between increased computational cost and improved prediction accuracy has become a challenge.

[0003] In typical attention mechanisms, the computational cost is O(n^2) because each element of the input sequence data is weighted. 2 (Incremented twice. "n" represents the length of the sequence data.)

[0004] As the computational load increases, it leads to increased inference time for models in large-scale data or real-time applications.

[0005] In addition, to improve prediction accuracy, in order for the attention mechanism to function properly, it is necessary to accurately grasp the hidden structural causal relationships in the sequence data and determine the appropriate computational range.

[0006] However, previous methods have not adequately grasped causal relationships and determined the scope of calculation, which sometimes limits the improvement of prediction accuracy.

[0007] Patent document 1 discloses a model acquisition method that improves model performance and reduces computational resource consumption. This method involves training the model according to a learning objective corresponding to the syntactic information of the self-attention module.

[0008] Existing technical documents

[0009] Patent documents

[0010] Patent Document 1: Japanese Patent No. 7379792 Summary of the Invention

[0011] The problem that the invention aims to solve

[0012] The purpose of this disclosure is to compute the feature values ​​of multiple elements shown in sequence data while suppressing the increase in computational cost.

[0013] Methods for solving problems

[0014] The information processing apparatus of this disclosure includes: a causal relationship derivation unit that takes sequence data representing a sequence of multiple elements as input and uses a causal relationship derivation model to deduce the causal relationship between the multiple elements; a block determination unit that, based on the deduced causal relationship, determines a block that becomes the computational scope of a self-attention mechanism for each of the elements; a self-attention calculation unit that takes the block of each of the multiple elements as the computational scope and calculates the self-attention value of each of the multiple elements using the self-attention mechanism; and a feature quantity calculation unit that takes the self-attention value of each of the multiple elements as input and uses an inference model to calculate the feature quantity of each of the multiple elements.

[0015] Invention Effects

[0016] According to this disclosure, chunks are determined based on the causal relationships among multiple elements shown in the sequence data. This allows for the computational efficiency of calculating the individual feature values ​​of the multiple elements shown in the sequence data. Attached Figure Description

[0017] Figure 1 This is a structural diagram of the information processing device 100 in Embodiment 1.

[0018] Figure 2 This is a structural diagram of the inference unit 110 in Embodiment 1.

[0019] Figure 3 This is a structural diagram of the learning unit 120 in embodiment 1.

[0020] Figure 4 This is a flowchart of the inference process in Implementation Method 1.

[0021] Figure 5 This is a diagram illustrating an example of causal relationship diagram 192 and matrix 193 in implementation method 1.

[0022] Figure 6 This is a diagram illustrating an example of matrix 193 in implementation method 1.

[0023] Figure 7 This is a diagram illustrating an example of parent-child node group 195 and block 194 in implementation method 1.

[0024] Figure 8 This is a diagram illustrating an example of block 194 in implementation method 1.

[0025] Figure 9 This is a diagram illustrating an example of block 194 in implementation method 1.

[0026] Figure 10 This is a diagram illustrating an example of block 194 in implementation method 1.

[0027] Figure 11 This is a flowchart illustrating the learning process in Implementation Method 1.

[0028] Figure 12 This is a flowchart illustrating the learning process in Implementation Method 1.

[0029] Figure 13 This is a hardware structure diagram of the information processing device 100 in Implementation Method 1. Detailed Implementation

[0030] In the embodiments and accompanying drawings, the same or corresponding elements are labeled with the same reference numerals. Descriptions of elements labeled with the same reference numerals as those already described are appropriately omitted or simplified. Arrows in the figures primarily indicate data flow or processing flow.

[0031] Implementation method 1.

[0032] based on Figures 1-13 This section explains how to calculate the individual feature values ​​of multiple elements in the sequence data while suppressing the increase in computational cost.

[0033] Structural description

[0034] based on Figure 1 This describes the structure of the information processing device 100.

[0035] The information processing device 100 is a computer equipped with hardware such as a processor 101, a memory 102, an auxiliary storage device 103, a communication device 104, and an input / output interface 105. These hardware components are interconnected via signal lines.

[0036] Processor 101 is an IC that performs computational processing and controls other hardware. For example, processor 101 is a CPU, DSP, or GPU.

[0037] IC is an abbreviation for Integrated Circuit.

[0038] CPU is an abbreviation for Central Processing Unit.

[0039] DSP is an abbreviation for Digital Signal Processor.

[0040] GPU is an abbreviation for Graphics Processing Unit.

[0041] Memory 102 is a volatile or non-volatile storage device. Memory 102 is also referred to as main storage device or main memory. For example, memory 102 is RAM. Data stored in memory 102 is stored in auxiliary storage device 103 as needed.

[0042] RAM is an abbreviation for Random Access Memory.

[0043] Auxiliary storage device 103 is a non-volatile storage device. For example, auxiliary storage device 103 is ROM, HDD, flash memory, or a combination thereof. Data stored in auxiliary storage device 103 is loaded into memory 102 as needed.

[0044] ROM is an abbreviation for Read Only Memory.

[0045] HDD is an abbreviation for Hard Disk Drive.

[0046] The communication device 104 is a receiver and a transmitter. For example, the communication device 104 is a communication chip or a NIC. The communication device 104 is used for communication with the information processing device 100.

[0047] NIC is an abbreviation for Network Interface Card.

[0048] Input / output interface 105 is a port for connecting input devices and output devices. For example, input / output interface 105 is a USB terminal, the input devices are a keyboard and mouse, and the output device is a monitor. The input and output of information processing device 100 are performed using input / output interface 105.

[0049] USB is an abbreviation for Universal Serial Bus.

[0050] The information processing device 100 includes elements such as an inference unit 110 and a learning unit 120. These elements are implemented through software.

[0051] The auxiliary storage device 103 stores an information processing program for enabling the computer to function as the inference unit 110 and the learning unit 120. The information processing program is loaded into the memory 102 and executed by the processor 101.

[0052] The OS is also stored in the auxiliary storage device 103. At least a portion of the OS is loaded into the memory 102 and executed by the processor 101.

[0053] The processor 101 executes information processing programs while executing the OS.

[0054] OS is an abbreviation for Operating System.

[0055] The data (input data, output data, etc.) of the information processing program are stored in the storage unit 190.

[0056] The memory 102 functions as the storage unit 190. However, auxiliary storage devices such as the auxiliary storage device 103, registers within the processor 101, and cache memory within the processor 101 may also replace the memory 102 or function as the storage unit 190 together with the memory 102.

[0057] Information processing programs can be recorded (stored) in a computer-readable manner on non-volatile recording media such as optical discs or flash memory.

[0058] based on Figure 2 The structure of the inference unit 110 will be explained.

[0059] The inference unit 110 includes a causal relationship deduction unit 111, a block determination unit 112, a self-attention calculation unit 113, and a feature quantity calculation unit 114.

[0060] Parameter data 191, sequence data 181, and inference data 182 are stored in storage unit 190.

[0061] based on Figure 3 The structure of the Study Department 120 will be explained.

[0062] The learning unit 120 includes a causal reasoning unit 121, a block decision unit 122, a self-attention calculation unit 123, a feature quantity calculation unit 124, and a learning control unit 125.

[0063] Learning data 183 and correct answer data 184 are stored in storage unit 190.

[0064] Description of the action

[0065] The steps of operating the information processing device 100 are equivalent to information processing methods. Furthermore, the steps of operating the information processing device 100 are equivalent to processing steps based on an information processing program.

[0066] based on Figure 4 The inference process of information processing methods is explained.

[0067] The parameters of the causal relationship inference model and the inference model after they have been learned are stored in parameter data 191.

[0068] For example, the parameters of the neural networks (NNs) used in each model are stored in parameter data 191.

[0069] The causal relationship derivation model and inference model will be discussed later.

[0070] In step S111, the causal relationship derivation unit 111 obtains the parameters of the causal relationship derivation model from the parameter data 191 and sets them in the causal relationship derivation model.

[0071] In addition, the feature calculation unit 114 obtains the parameters of the inference model from the parameter data 191 and sets them in the inference model.

[0072] In step S112, the causal relationship inference unit 111 obtains sequence data 181.

[0073] For example, the user inputs sequence data 181 into the information processing device 100, and the causal relationship prediction unit 111 receives the input sequence data 181.

[0074] Sequence data is data that represents a sequence of multiple elements.

[0075] For example, sequence data can represent multiple values ​​arranged in a time series or multiple words arranged in an article.

[0076] Examples of sequence data include natural language article data or sensor data from FA (Factory Automation). FA is an abbreviation for factory automation.

[0077] Then, the causal relationship derivation unit 111 takes the sequence data 181 as input and uses the causal relationship model to deduce the causal relationship between multiple elements.

[0078] Therefore, causal relationship diagram 192 was created to represent the inferred causal relationship.

[0079] Explain the causal relationship model.

[0080] A causal relationship model is a machine learning model that derives the causal relationships between multiple elements shown in the input data. The causal relationship model is stored in storage unit 190.

[0081] Causal relationship models, for example, derive causal relationships using convex optimization methods.

[0082] The causal relationship model outputs causal relationship graph 192, which serves as data representing the inferred causal relationships.

[0083] The causal relationship diagram 192 represents the causal relationship of a linear structure of multiple elements.

[0084] Specifically, the causal relationship graph 192 is represented by taking multiple elements as nodes and connecting nodes with direct causal relationships to each other with paths.

[0085] The causal relationship diagram 192 includes path coefficients for each path.

[0086] Path coefficients represent the magnitude of the causal relationship between nodes connected by a path.

[0087] The causal relationship diagram 192 can be represented by matrix 193.

[0088] Figure 5 Examples of causal relationship diagrams 192 and 193 are shown.

[0089] Sequence data 181 shows a sequence of 9 elements. These 9 elements are extracted, for example, using a neural network.

[0090] The causal relationship diagram 192 has 9 nodes (quadrilaterals) representing 9 elements.

[0091] In causal relationship diagram 192, nodes with direct causal relationships are connected to each other by paths (arrows).

[0092] Each path is assigned a path coefficient, which represents the magnitude of the causal relationship between the nodes connected by the path.

[0093] In a path connecting two nodes, one node (the starting side of the arrow) is called the parent node, and the other node (the ending side of the arrow) is called the child node.

[0094] For example, the sixth element node has direct causal relationships with the third, fifth, eighth, and ninth element nodes. The third element node is the parent node of the sixth element. The fifth, eighth, and ninth element nodes are the child nodes of the sixth element.

[0095] Matrix 193 has 9 queries (rows) with 9 identical elements and 9 keys (columns) with 9 identical elements. The attention, described later, is calculated using the keys that become the parent or child nodes of the queries.

[0096] The shaded areas in matrix 193 represent groups of nodes with a direct causal relationship.

[0097] Figure 6 An example representing matrix 193.

[0098] Matrix 193 can be permuted into matrix 193I and matrix 193T.

[0099] Matrix 193I is obtained by adding the identity matrix to matrix 193.

[0100] Matrix 193T is obtained by adding the transpose matrix and the identity matrix to matrix 193.

[0101] return Figure 4 The explanation will continue from step S113.

[0102] In step S113, the block determination unit 112 determines the block 194 of each element based on the derived causal relationship.

[0103] Block 194 is the computational scope of the self-attention mechanism.

[0104] Specifically, the block decision unit 112 selects a parent-child node group 195 from the causal relationship graph 192 according to each element, and determines the block 194 based on the parent-child node group 195 according to each element.

[0105] The parent-child node group 195 consists of an object element node representing an object element and nodes that are in a parent-child relationship with the object element node.

[0106] For example, block decision unit 112 determines block 194 based on parent-child node group 195 as follows.

[0107] The block determination unit 112 determines the element group corresponding to the parent-child node group 195 as block 194.

[0108] Figure 7 This represents an example of parent-child node group 195 and block 194.

[0109] The parent-child node group 195 of the sixth element consists of the third element node, the fifth element node, the sixth element node, the eighth element node, and the ninth element node.

[0110] In this case, the third, fifth, sixth, eighth, and ninth elements corresponding to the parent-child node group of the sixth element become block 194 of the sixth element. The length of block 194 of the sixth element is 5.

[0111] return Figure 4 Continuing with the explanation of step S113.

[0112] For example, block decision unit 112 determines block 194 based on parent-child node group 195 as follows.

[0113] The block decision unit 112 compares the number of nodes in the parent-child node group 195 with a constant. The constant is a predetermined fixed value.

[0114] When the number of nodes in parent-child node group 195 is the same as the constant, the block determination unit 112 determines the element group corresponding to parent-child node group 195 as block 194.

[0115] If the number of nodes in the parent-child node group 195 is less than a constant, the block determination unit 112 determines the element group corresponding to the parent-child node group 195 and one or more virtual elements as block 194.

[0116] If the number of nodes in the parent-child node group 195 exceeds a constant, the block determination unit 112 determines the same number of elements as the constant from the element group corresponding to the parent-child node group 195 as block 194. Specifically, the block determination unit 112 selects the same number of elements as the constant from the element group corresponding to the parent-child node group 195 as block 194 based on the path coefficient group for the parent-child node group 195.

[0117] Figure 8 An example of block 194 is shown.

[0118] The parent-child node group 195 of the first element consists of the first element node, the seventh element node, the fourth element node, and the second element node.

[0119] The number of nodes in the parent-child node group 195 of the first element is 4. With a constant of 4, the number of nodes in the parent-child node group 195 of the first element is the same as the constant.

[0120] In this case, the first element, the seventh element, the fourth element, and the second element are block 194 of the first element.

[0121] In addition, in block 194, the remaining elements (seventh element, fourth element, and second element) except for the object element (first element) are sorted in descending order of path coefficient.

[0122] Figure 9 An example of block 194 is shown.

[0123] The parent-child node group 195 of the second element consists of the second element node, the third element node, and the first element node.

[0124] The number of nodes in the parent-child node group 195 of the second element is 3. When the constant is 4, the number of nodes in the parent-child node group 195 of the second element is less than the constant.

[0125] In this case, the second element, the third element, the first element, and the 0th element are block 194 of the second element. The 0th element is a virtual element that supplements block 194.

[0126] In addition, in block 194, the remaining elements (first and third elements) except for the object element (second element) and the virtual element (0th element) are sorted in descending order of path coefficient.

[0127] Figure 10 An example of block 194 is shown.

[0128] The parent-child node group 195 of the sixth element consists of the sixth element node, the ninth element node, the eighth element node, the third element node, and the fifth element node.

[0129] The parent-child node group 195 of the sixth element has 5 nodes. With a constant of 4, the number of nodes in the parent-child node group 195 of the sixth element exceeds the constant.

[0130] In this case, the parent-child node group 195 of the sixth element is sorted in descending order of path coefficient. Then, the same number of elements as the constant are selected in descending order of path coefficient, and these selected elements form block 194 of the sixth element. Block 194 of the sixth element consists of the sixth element, the ninth element, the eighth element, and the third element. The fifth element is not included in block 194 of the sixth element. That is, the fifth element is not used.

[0131] return Figure 4 The explanation will continue from step S114.

[0132] In step S114, the self-attention calculation unit 113 uses the blocks 194 of each of the multiple elements as the calculation range and calculates the self-attention value of each of the multiple elements through the self-attention mechanism.

[0133] The self-attention mechanism is implemented through an algorithm used to calculate the value of self-attention.

[0134] In step S115, the feature calculation unit 114 takes the self-attention values ​​of each of the multiple elements as input and uses the inference model to calculate the feature values ​​of each of the multiple elements.

[0135] Therefore, inference data 182 is created as data representing the characteristic quantities of multiple elements.

[0136] The inference model is explained.

[0137] An inference model is a completed machine learning model that computes multiple features corresponding to the self-attention values ​​shown in the input data and calculates inference results based on these features. These features are also called hidden features. The inference results vary depending on the task, representing predictions, classifications, etc.

[0138] The inference model outputs inference data 182, which serves as data representing multiple feature quantities and inference results.

[0139] based on Figure 11 and Figure 12 This section explains the learning process of information processing methods.

[0140] In step S121, the causal relationship derivation unit 121 initializes the parameters of the causal relationship derivation model.

[0141] In addition, the feature calculation unit 114 initializes the parameters of the inference model.

[0142] In step S122, the causal relationship prediction unit 121 obtains learning data 183.

[0143] For example, the user inputs learning data 183 into the information processing device 100, and the causal relationship prediction unit 111 receives the input learning data 183.

[0144] Learning data 183 is sequence data used for learning.

[0145] Then, the causal relationship derivation unit 111 takes the sequence of multiple elements for learning shown in the learning data 183 as input and uses the causal relationship model to deduce the causal relationship of the multiple elements for learning.

[0146] Therefore, causal relationship diagram 192 was created as data representing the inferred causal relationships.

[0147] In step S123, the block determination unit 122 determines the block 194 of each element based on the derived causal relationship. The determination method is the same as that in step S113.

[0148] In step S124, the self-attention calculation unit 123 uses the blocks 194 of each of the multiple elements to be learned as the calculation range, and calculates the self-attention value of each of the multiple elements to be learned through the self-attention mechanism.

[0149] In step S125, the feature calculation unit 124 takes the self-attention values ​​of each of the multiple elements to be learned as input and uses the inference model to calculate the feature values ​​of each of the multiple elements to be learned. As a result, inference data 182 is generated.

[0150] In step S126, the learning control unit 125 determines whether the learning has converged based on the feature quantities of each of the multiple elements calculated in step S125.

[0151] Determine the convergence of learning as follows.

[0152] First, the learning control unit 125 obtained 184 correct solution data.

[0153] For example, the user inputs the correct solution data 184 into the information processing device 100, and the learning control unit 125 receives the input correct solution data 184.

[0154] The positive solution data 184 represents the positive solutions for multiple feature quantities and the positive solutions for inference results.

[0155] Next, the learning control unit 125 compares the inferred data 182 with the correct answer data 184.

[0156] Then, the learning control unit 125 determines whether the learning has converged based on the comparison results.

[0157] For example, the learning control unit 125 calculates the feature difference. The feature difference represents the magnitude of the difference between multiple features shown in the inferred data 182 and multiple features shown in the positive solution data 184. Then, the learning control unit 125 compares the feature difference with a threshold, and determines that the learning has converged if the feature difference is less than the threshold.

[0158] If the learning converges, proceed to step S127.

[0159] If the learning process fails to converge, the learning control unit 125 updates the parameters of the causal relationship prediction model and the inference model based on the comparison results. Afterwards, the process proceeds to step S122.

[0160] In step S127, the causal relationship derivation unit 121 generates sequence data that represents the sequence of multiple feature quantities corresponding to multiple elements for learning as a new sequence of multiple elements for learning.

[0161] Then, the causal relationship derivation unit 121 takes the generated sequence data as input and uses the causal relationship derivation model to derive the causal relationships of the new multiple elements for learning.

[0162] In step S128, the learning control unit 125 determines whether the difference between the causal relationship derived in step S127 and the causal relationship derived in step S122 is large.

[0163] Determine the magnitude of the difference between causal relationships as follows.

[0164] First, the learning control unit 125 updates the causal relationship graph 192 created in step S122 to the causal relationship graph 192 created in step S127, and calculates the graph update amount. The graph update amount represents the amount of update from the causal relationship graph 192 created in step S122 to the causal relationship graph 192 created in step S127.

[0165] Then, the learning control unit 125 compares the graph update amount with the threshold. If the graph update amount is greater than the threshold, it determines that the difference between the causal relationships is large.

[0166] When the differences between causal relationships are large, the learning control unit 125 updates the parameters of the causal relationship prediction model and the inference model based on the comparison results. Then, the process proceeds to step S122.

[0167] If the differences between the causal relationships are not significant, proceed to step S129.

[0168] In step S129, the causal relationship derivation unit 111 saves the parameters of the causal relationship derivation model in parameter data 191.

[0169] In addition, the feature calculation unit 114 stores the parameters of the inference model in the parameter data 191.

[0170] The learning process may also omit steps S127 and S128. In this case, if convergence is achieved in step S126, the process proceeds to step S129.

[0171] Effects of Implementation Method 1

[0172] Implementation 1 determines the range of feature quantity calculations based on causal relationships during the learning and inference of a prediction model that uses an attention mechanism, thereby improving the accuracy of the prediction model.

[0173] Implementation Method 1: In order to simultaneously reduce the computational cost of the attention mechanism and improve the prediction accuracy, a new method is proposed.

[0174] By combining causal reasoning, self-attention block determination, and self-attention value calculation, the system can appropriately control the computational range, and the performance of the prediction model can be expected to be improved.

[0175] In models that use attention mechanisms, accuracy may decrease if the correlation of features is insufficient.

[0176] Implementation Method 1: Derive the cause-effect graph based on the input data, and appropriately save and utilize the learned parameters, thereby improving the performance of the model during learning and inference.

[0177] In Implementation 1, the causal relationship derivation unit and the hidden feature calculation unit are appropriately learned during learning. During inference, these learned parameters are read from the database to derive causal relationships and calculate feature quantities. As a result, the performance of the model can be improved.

[0178] According to Implementation Method 1, the range of automatic feature quantity calculation based on causal relationships can be determined during model learning and inference, thereby improving the prediction accuracy of the model.

[0179] Furthermore, by learning the linear structure of the causal relationship derivation unit and the hidden feature calculation unit, and appropriately calculating the model's feature values, the performance of the prediction model can be improved.

[0180] Summary of Implementation Method 1

[0181] The information processing device 100 is a system that includes a causal relationship deduction unit 111, a block determination unit 112, a self-attention calculation unit 113, and a feature quantity calculation unit 114. These structures cooperate to automatically and efficiently calculate the feature quantities.

[0182] The causal relationship derivation unit 111 derives the causal graph based on the input data.

[0183] Based on the obtained causal graph, the block decision unit 112 determines the scope that can elucidate the direct causal relationships of the data.

[0184] The self-attention calculation unit 113 calculates the value of self-attention based on this range.

[0185] The feature calculation unit 114 extracts hidden features through machine learning.

[0186] By incorporating these structures into a system, beneficial information containing self-attention can be automatically and efficiently obtained from the input data, enabling the construction of high-level predictive models.

[0187] To determine the range of the self-attention block, a convex optimization method is used to determine the causal relationship of the input data.

[0188] When determining the scope of a block, select data with a direct causal relationship and determine the scope of the block based on that data.

[0189] The block length is set to a constant. When the block length is less than the constant, nodes are sorted in descending order of path coefficients in the causal graph, and any missing nodes are filled with nodes of size 0. When the block length exceeds the constant, nodes are sorted in descending order of path coefficients in the causal graph, and any excess nodes are not included in the block.

[0190] By combining convex optimization methods with block decision methods that utilize causal data, and appropriately controlling the computational scope, efficient and effective self-attention computation can be achieved.

[0191] Supplement to Implementation Method 1

[0192] By implementing method 1, the following advantages can be achieved, and the performance of the prediction model can be expected to be improved.

[0193] (1) Consideration of correlation and judgment of direct causal relationship

[0194] To consider the correlation of the features used for prediction, we need to determine the direct causal relationships between nodes in the input sequence data.

[0195] To compute the features of self-attention, the computation range (block) is automatically determined based on the input data (key, query) of length n.

[0196] (2) Improved prediction accuracy

[0197] The structural causal relationships of the input sequence data (key, query) are derived, and the scope of the blocks is appropriately determined based on the parent-child relationships of these causal relationships. Thus, the model captures important correlations and can be expected to achieve high prediction accuracy.

[0198] (3) The amount of calculation required for inference is reduced.

[0199] By appropriately determining the scope (block) of computation based on causal reasoning, the computational load during inference can be significantly reduced.

[0200] Specifically, the computational complexity ranges from O(n^2) to O(n^2). 2 Reduced to O(n·k). “k” represents the block size.

[0201] Therefore, it enables efficient inference, improves real-time performance, and can adapt to large-scale data.

[0202] By leveraging these strengths, and appropriately capturing the features of the input sequence data, a prediction model that takes into account the correlation between nodes can be constructed, thereby improving prediction accuracy.

[0203] based on Figure 13 The hardware structure of the information processing device 100 will be described.

[0204] The information processing device 100 includes a processing circuit 109.

[0205] The processing circuit 109 is the hardware that implements the inference unit 110 and the learning unit 120.

[0206] The processing circuit 109 can be dedicated hardware or a processor 101 that executes the program stored in the memory 102.

[0207] When the processing circuit 109 is dedicated hardware, the processing circuit 109 may be, for example, a single circuit, a composite circuit, a programmable processor, a parallel programmable processor, an ASIC, an FPGA, or a combination thereof.

[0208] ASIC is an abbreviation for Application Specific Integrated Circuit.

[0209] FPGA is an abbreviation for Field Programmable Gate Array.

[0210] The information processing device 100 may also have multiple processing circuits that replace the processing circuit 109.

[0211] In the processing circuit 109, some functions may be implemented by dedicated hardware, while the remaining functions may be implemented by software or firmware.

[0212] In this way, the functions of the information processing device 100 can be implemented through hardware, software, firmware, or a combination thereof.

[0213] Embodiment 1 is an example of a preferred embodiment and is not intended to limit the scope of this disclosure. Embodiment 1 may be implemented in part or in combination with other embodiments. The steps described using flowcharts or the like may also be appropriately modified.

[0214] The term "component" in the various elements of the information processing device 100 can also be replaced with "processing", "process", "circuit" or "line".

[0215] The various methods disclosed herein are hereby recorded as supplementary notes.

[0216] (Note 1)

[0217] An information processing apparatus comprising: The causal relationship derivation unit takes sequence data representing a sequence of multiple elements as input and uses a causal relationship derivation model to deduce the causal relationship between the multiple elements. The block-determining unit, based on the derived causal relationships, determines the blocks that become the computational scope of the self-attention mechanism according to each of the elements; A self-attention calculation unit, which uses the blocks of each of the plurality of elements as the calculation range, calculates the self-attention value of each of the plurality of elements through the self-attention mechanism; and The feature calculation unit takes the value of the self-attention of each of the plurality of elements as input and uses an inference model to calculate the feature of each of the plurality of elements.

[0218] (Note 2)

[0219] According to the information processing apparatus described in Appendix 1, wherein, The causal relationship derivation model is a model that derives causal relationships using convex optimization methods.

[0220] (Note 3)

[0221] According to the information processing apparatus described in Appendix 1 or 2, wherein, The causal relationship derivation unit creates a causal relationship graph, which represents the multiple elements as nodes and connects nodes with direct causal relationships to each other using paths. This graph serves as data representing the deduced causal relationships. The block determination unit selects, for each element, a node representing the element and a node in a parent-child relationship with the node of the element from the causal relationship graph as a parent-child node group, and determines the block based on the parent-child node group for each element.

[0222] (Note 4)

[0223] According to the information processing apparatus described in Appendix 3, wherein... The block determination unit determines the group of elements corresponding to the parent-child node group as the block.

[0224] (Note 5)

[0225] According to the information processing apparatus described in Appendix 3, wherein... The block decision unit If the number of nodes in the parent-child node group is the same as a constant, the element group corresponding to the parent-child node group will be determined as the block. If the number of nodes in the parent-child node group is less than the constant, the element group corresponding to the parent-child node group and one or more virtual elements will be determined as the block. If the number of nodes in the parent-child node group exceeds the constant, the same number of elements in the element group corresponding to the parent-child node group that is equal to the constant will be determined as the block.

[0226] (Note 6)

[0227] According to the information processing apparatus described in Appendix 5, wherein... The causal graph includes path coefficients for each path, representing the magnitude of the causal relationship between the nodes connected by the path. When the number of nodes in the parent-child node group exceeds the constant, the block determination unit selects the same number of elements as the constant from the element group corresponding to the parent-child node group as the block, based on the path coefficient group for the parent-child node group.

[0228] (Note 7)

[0229] An information processing method, wherein, Taking sequence data representing a sequence of multiple elements as input, a causal relationship derivation model is used to deduce the causal relationships among the multiple elements. Based on the derived causal relationships, the blocks that constitute the computational scope of the self-attention mechanism are determined according to each of the aforementioned elements. Using the blocks of each of the plurality of elements as the computational scope, the self-attention values ​​of each of the plurality of elements are calculated through the self-attention mechanism. Using the self-attention values ​​of each of the plurality of elements as input, the feature quantities of each of the plurality of elements are calculated using an inference model.

[0230] (Postscript 8)

[0231] An information processing program that causes a computer to perform the following processes: The causal relationship derivation process takes sequence data representing a sequence of multiple elements as input and uses a causal relationship derivation model to deduce the causal relationship between the multiple elements. The block decision process, based on the derived causal relationship, determines which block becomes the computational scope of the self-attention mechanism according to each element. Self-attention computation processing, taking the blocks of each of the plurality of elements as the computation range, and calculating the self-attention value of each of the plurality of elements through the self-attention mechanism; and The feature calculation process takes the values ​​of the self-attention of each of the plurality of elements as input and uses an inference model to calculate the feature values ​​of each of the plurality of elements.

[0232] Label Explanation

[0233] 100: Information processing device; 101: Processor; 102: Memory; 103: Auxiliary storage device; 104: Communication device; 105: Input / output interface; 109: Processing circuit; 110: Inference unit; 111: Causal relationship derivation unit; 112: Block decision unit; 113: Self-attention calculation unit; 114: Feature quantity calculation unit; 120: Learning unit; 121: Causal relationship derivation unit; 122: Block decision unit; 123: Self-attention calculation unit; 124: Feature quantity calculation unit; 181: Sequence data; 182: Inference data; 183: Learning data; 184: Forward solution data; 190: Storage unit; 191: Parameter data; 192: Causal relationship graph; 193: Matrix; 194: Block; 195: Parent-child node group.

Claims

1. An information processing apparatus, comprising: The causal relationship derivation unit takes sequence data representing a sequence of multiple elements as input and uses a causal relationship derivation model to deduce the causal relationship between the multiple elements. The block-determining unit, based on the derived causal relationships, determines the blocks that become the computational scope of the self-attention mechanism according to each of the elements; The self-attention calculation unit takes the blocks of each of the plurality of elements as the calculation range and calculates the self-attention value of each of the plurality of elements through the self-attention mechanism. as well as The feature calculation unit takes the value of the self-attention of each of the plurality of elements as input and uses an inference model to calculate the feature of each of the plurality of elements.

2. The information processing apparatus according to claim 1, wherein, The causal relationship derivation model is a model that derives causal relationships using convex optimization methods.

3. The information processing apparatus according to claim 1 or 2, wherein, The causal relationship derivation unit creates a causal relationship graph, which represents the multiple elements as nodes and connects nodes with direct causal relationships to each other using paths. This graph serves as data representing the deduced causal relationships. The block determination unit selects, for each element, a node representing the element and a node in a parent-child relationship with the node of the element from the causal relationship graph as a parent-child node group, and determines the block based on the parent-child node group for each element.

4. The information processing apparatus according to claim 3, wherein, The block determination unit determines the group of elements corresponding to the parent-child node group as the block.

5. The information processing apparatus according to claim 3, wherein, The block decision unit If the number of nodes in the parent-child node group is the same as a constant, the element group corresponding to the parent-child node group will be determined as the block. If the number of nodes in the parent-child node group is less than the constant, the element group corresponding to the parent-child node group and one or more virtual elements will be determined as the block. If the number of nodes in the parent-child node group exceeds the constant, the same number of elements in the element group corresponding to the parent-child node group that is equal to the constant will be determined as the block.

6. The information processing apparatus according to claim 5, wherein, The causal graph includes path coefficients for each path, representing the magnitude of the causal relationship between the nodes connected by the path. When the number of nodes in the parent-child node group exceeds the constant, the block determination unit selects the same number of elements as the constant from the element group corresponding to the parent-child node group as the block, based on the path coefficient group for the parent-child node group.

7. An information processing method, wherein, Taking sequence data representing a sequence of multiple elements as input, a causal relationship derivation model is used to deduce the causal relationships among the multiple elements. Based on the derived causal relationships, the blocks that constitute the computational scope of the self-attention mechanism are determined according to each of the aforementioned elements. Using the blocks of each of the plurality of elements as the computational scope, the self-attention values ​​of each of the plurality of elements are calculated through the self-attention mechanism. Using the self-attention values ​​of each of the plurality of elements as input, the feature quantities of each of the plurality of elements are calculated using an inference model.

8. An information processing program that causes a computer to perform the following processes: The causal relationship derivation process takes sequence data representing a sequence of multiple elements as input and uses a causal relationship derivation model to deduce the causal relationship between the multiple elements. The block decision process, based on the derived causal relationship, determines which block becomes the computational scope of the self-attention mechanism according to each element. Self-attention computation processing takes the blocks of each of the multiple elements as the computation range, and calculates the self-attention value of each of the multiple elements through the self-attention mechanism; as well as The feature calculation process takes the values ​​of the self-attention of each of the plurality of elements as input and uses an inference model to calculate the feature values ​​of each of the plurality of elements.