Information processing device, information processing method, and information processing program

Abstract

An information processing device 1 comprises an input unit 11 that inputs a plurality of time series data measured using a plurality of sensors at different positions, and a processing unit 12 that divides each of the plurality of time series data into a plurality of partial time series data at prescribed time intervals, generates a plurality of primary capsules each including feature vectors of the plurality of partial time series data related to the plurality of time series data, carries out graph modeling for generating a weighted matrix which uses weightings corresponding to the prescribed time intervals and the distances between the plurality of sensors to represent the linking relationship among the plurality of primary capsules, and carries out graph Fourier transforms of the respective feature vectors of the plurality of primary capsules based on the weighting matrix.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] By utilizing IoT and AI, it is possible to collect huge amounts of data from multiple types of sensors. For example, it is possible to collect a wide variety of data, such as road image data measured by in-vehicle cameras, vehicle position data measured by GPS, paddy field soil data measured by drones, and rice cultivation data measured by sensors.

[0003] On the other hand, the relationships between data are becoming more complex and difficult to understand. On the other hand, with the advancement of smart cities and autonomous driving technology, it is necessary to simultaneously analyze various types of data using the same analytical method and to implement measures according to the analysis results. Therefore, technology that performs linked learning and inference while taking into account the relationships between data is becoming increasingly important.

[0004] Traditionally, one type of machine learning used for data analysis is deep learning, which has deeply multi-layered layers of neurons. For example, there is the Convolutional Neural Network (CNN). CNN is a deep learning model that has a convolutional layer that extracts image features, a pooling layer that compresses the dimensions of the extracted features, and a connection layer that classifies images. Adjusting the degree of connection (weights) between neurons so that the output result value matches the input image is called learning, and judging the input image using the CNN after adjusting the weights is called inference.

[0005] However, in CNN, the neurons themselves, which are the learning units, lack the ability to accurately represent the features in the data. In particular, because they use pooling layers, the spatial relationship between multiple feature elements is lost, making the relationship between the features in the data and the inference results unclear. In other words, CNN lacks the interpretability of learning from data, making it difficult for humans to understand, making it difficult to realize HITL (Human In The Loop), which involves humans intervening in some judgments.

[0006] Therefore, Non-Patent Document 1 discloses a Capsule Network as a model capable of recognizing and recording feature elements in data. In CNN, feature elements in data are input and output as scalars, but in the Capsule Network, they are input and output as vectors. Feature elements in data are stored on each axis of a vector. In the case of a Capsule Network that executes the task of "recognizing the presence of a line", for example, information on the thickness of the line is stored on the first axis, and the direction of the line is stored on the second axis. A capsule refers to a state in which information on multidimensional feature elements in units of pixels or objects, etc., is vectorized and stored.

[0007] Capsule networks differ from CNNs in that (1) they use vectors instead of scalars, (2) they add processing to give spatial relationships to feature elements in the data, (3) they use dynamic routing instead of backpropagation when weighting the output vector, and (4) they use squash functions instead of sigmoid functions as activation functions. Below, we will outline the processing steps of capsule networks.

[0008] Step 1. Perform convolution on the image data to obtain a multidimensional vector u i (See Fig. 6 in Non-Patent Document 1 and "Algorithm 1: Capsule Networks" in Non-Patent Document 2.) i is called the primary capsule, and this layer is called the primary capsule layer. The primary capsule vector u iis, for example, the vector of the capsule corresponding to the hand of the human body.

[0009] Step 2. Primary capsule vector u i For the vector u i The weight matrix W corresponding to ij Multiply by vector u j|i (See Non-Patent Document 2 above) to obtain a weight indicating the spatial relationship of the hand to the human body. Steps 1 and 2 are also performed on the vectors of each capsule of the feet, etc.

[0010] Step 3. Each vector u j|i For each weight c ij Multiply and add up the sum, and then use Σ i c ij u j|i Let s be the input vector to the next layer. j (See Non-Patent Document 2 above.) The weight c ij is calculated during capsule network training using a method called dynamic routing.

[0011] Step 4. In the next layer, the input vector s j Apply the squash function to the output vector v j (See Non-Patent Document 2 above.) The output vector v j is the vector of the capsule of the human body that takes into account the spatial relationship of each part. The squash function is j This activation function scales the length of the output vector v within the range of 0 to 1 while maintaining its orientation. j is called a digit capsule, and this layer is called the digit capsule layer.

[0012] Here, the weight c ij is the vector u i is the degree of connectivity that represents how connected the whole is, and is calculated by using the softmax function ij = {exp(b ij )} / {Σ kexp(b ik )}" is calculated. ij The calculation method for optimizing the value of the vector u during learning is dynamic routing. i Dynamic routing is a method of calculating the importance of each capsule when generating a high-level capsule from a low-level capsule in a previous layer.

[0013] where b ij The value of "b ij ←b ij +u j|i ·v j " will be updated. j|i ·v j " is called the agreement calculation, and is a formula for calculating the degree of agreement between the vector of a primary capsule in the previous layer and the vector of a digit capsule in the next layer.

[0014] However, agreement calculation requires a lot of computational resources and time, making it inefficient. j|i or j The weight matrix W used in the calculation of ij The number of parameters is huge. The number of parameters is two to three times that of CNN, so b ij The time it takes for the value of to converge is 10 to 30 times longer than that of CNN. As a result, the output vector v j is calculated, and it may not be possible to properly separate each feature element.

[0015] Therefore, Non-Patent Document 2 discloses a graph capsule network, in which a calculation method of dynamic routing, which requires a large amount of calculation, is replaced with a calculation method of multi-head attention-based graph pooling (see FIG. 1 in Non-Patent Document 2).

[0016] The processing procedure of the graph capsule network is different from the above capsule network in steps 2 and 3. The multidimensional vector u obtained in step 1 is i ,K 2 The nodes of the graph are transformed with identity. Then, the transformed vector u i ' is multiplied by the weight matrix A, which represents the weight (=attention) between adjacent nodes, to obtain the vector u i Then, for each feature vector u i The capsule group of ' is dimensionally reduced to the input vector s j get. [Prior art documents] [Non-patent literature]

[0017] [Non-Patent Document 1] Rinat Mukhometzianov and 1 others, “CapsNet comparative performance evaluation for image classification” [Non-Patent Document 2] Jindong Gu, 1 others, “Interpretable Graph Capsule Networks for Object Recognition”, arXiv:2012.01674v3 [cs.CV] 7 Mar 2021 Summary of the Invention [Problem to be solved by the invention]

[0018] However, since Non-Patent Document 2 is merely a technology for encapsulating image data having feature elements in space, it is difficult to apply it to time-series data having feature elements on the time axis.In addition, Non-Patent Document 2 does not disclose a mechanism for separating each feature element of the time-series data, so the output vector is calculated while the feature elements in the time-series data remain intertwined with each other.

[0019] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a technique capable of improving the classification accuracy of each feature element contained in multidimensional time-series data having feature elements in time and space. [Means for solving the problem]

[0020] An information processing device of one embodiment of the present invention includes an input unit that inputs a plurality of time series data measured by a plurality of sensors at different positions, and a processing unit that divides the plurality of time series data into a plurality of partial time series data at a predetermined time interval, generates a plurality of primary capsules each including a feature vector of the plurality of partial time series data related to the plurality of time series data, performs graph modeling to generate a weighting matrix that represents the connection relationship between the plurality of primary capsules with weights according to the distance between the plurality of sensors and the predetermined time interval, and performs a graph Fourier transform of each feature vector of the plurality of primary capsules based on the weighting matrix.

[0021] An information processing device of one embodiment of the present invention includes an input unit that inputs a plurality of time series data measured by a plurality of sensors at different positions, and a processing unit that uses the plurality of time series data to separately generate a plurality of primary capsules each including a feature vector for each of the time series data, and a plurality of primary capsules each including a feature vector for each predetermined time interval.

[0022] An information processing method of one aspect of the present invention is an information processing method performed by an information processing device, which includes the steps of inputting a plurality of time series data measured by a plurality of sensors at different positions, dividing the plurality of time series data into a plurality of partial time series data at a predetermined time interval, generating a plurality of primary capsules each including each feature vector of the plurality of partial time series data related to the plurality of time series data, performing graph modeling to generate a weighting matrix that represents the connection relationship between the plurality of primary capsules with weights corresponding to the distance between the plurality of sensors and the predetermined time interval, and performing a graph Fourier transform of each feature vector of the plurality of primary capsules based on the weighting matrix.

[0023] An information processing method according to one embodiment of the present invention is an information processing method performed by an information processing device, which includes the steps of inputting a plurality of time series data measured by a plurality of sensors at different positions, and using the plurality of time series data to separately generate a plurality of primary capsules each including a feature vector for each of the time series data, and a plurality of primary capsules each including a feature vector for each predetermined time interval.

[0024] An information processing program according to one aspect of the present invention causes a computer to function as the information processing device. Effect of the Invention

[0025] According to the present invention, it is possible to provide a technique capable of improving the classification accuracy of each feature element included in multidimensional time series data having feature elements in time and space. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a diagram illustrating a functional block configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram showing a processing flow of the information processing device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram showing a processing image of the information processing device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an image of division of a time-series data group. [Diagram 5] FIG. 5 is a diagram showing an image of encapsulation. [Figure 6] FIG. 6 is a diagram showing an image of graph modeling. [Figure 7] FIG. 7 is a diagram showing an image of the graph Fourier transform. [Figure 8] FIG. 8 is a diagram showing an image of filtering. [Figure 9] FIG. 9 is a diagram showing an image of attention calculation. [Figure 10]FIG. 10 is a diagram showing an image of classification of signal values. [Figure 11] FIG. 11 is a diagram illustrating a processing flow of the information processing device according to the second embodiment. [Figure 12] FIG. 12 is a diagram showing a processing image of the information processing device according to the second embodiment. [Figure 13] FIG. 13 is a diagram illustrating a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are given the same reference numerals and the description will be omitted.

[0028] [Summary of the Invention] In order to solve the above problems, the present invention discloses two embodiments.

[0029] In the first embodiment, a method for generating a capsule in which feature elements in time and space are simultaneously graph-modeled for multidimensional time series data having feature elements in time and space is disclosed. In particular, in order to be applicable not only to feature elements on the spatial axis but also to feature elements on the time axis, graph modeling is performed using a weight matrix based on both the spatial axis and the time axis. In addition, in order to improve the classification accuracy of each feature element included in the time series data, a graph Fourier transform is performed to convert feature elements in the time and space domain into the spectral domain. This makes it possible to apply the method to multidimensional time series data having feature elements on the time axis, and to completely separate each feature element from the other feature elements.

[0030] In the second embodiment, a method is disclosed for encapsulating feature elements on the space axis and the time axis separately for multidimensional time series data having feature elements in time and space. Since feature elements are encapsulated separately on the space axis and the time axis, the method is applicable to multidimensional time series data having feature elements on the time axis, and each feature element can be completely separated from the other feature elements.

[0031] [First embodiment] 1 is a diagram showing a functional block configuration of an information processing device 1 according to the first embodiment. The information processing device 1 includes an input unit 11, a processing unit 12, an output unit 13, and a storage unit .

[0032] The input unit 11 has a function of inputting a plurality of time series data measured by a plurality of sensors at different positions.

[0033] The processing unit 12 has a function of dividing the multiple time series data into multiple partial time series data at a predetermined time interval, generating multiple primary capsules each including each feature vector of the multiple partial time series data related to the multiple time series data, performing graph modeling to generate a weighting matrix that represents the connection relationship between the multiple primary capsules with weights according to the distances between the multiple sensors and the predetermined time intervals, and performing a graph Fourier transform of each feature vector of the multiple primary capsules based on the weighting matrix.

[0034] The processing unit 12 has a function of generating multiple capsules in the spectral domain by classifying multiple signal values ​​contained in multiple bases obtained by the graph Fourier transform according to signal values ​​at the same position within the base, and calculating the attention of the classification class based on the magnitude of the signal values ​​contained in the multiple capsules.

[0035] The output unit 13 has a function of outputting the inference result of the task inferred based on the calculation result of the attention of the classification class.

[0036] The storage unit 14 has a function of storing any data used during learning or inference. For example, the storage unit 14 stores a plurality of time-series data input to the information processing device 1, a learning model used during learning or inference, various parameters of the learning model, and the like.

[0037] Fig. 2 is a diagram showing a processing flow of the information processing device 1 according to the first embodiment. Fig. 3 is a diagram showing a processing image of the information processing device 1 according to the first embodiment.

[0038] Step S101; The input unit 11 receives m pieces of time series data X{x1, x2, ..., x m}(x i ∈R l ) is input. m is the total number of time series data. R is the real number space. l is the time series data x i where l is the time length. l is also the total number of time stamps (measurement times). The time series data is, for example, time series data of electrocardiogram values ​​measured by sensor 1, time series data of electroencephalogram values ​​measured by sensor 2, ..., time series data measured by sensor m. The time series data handled in the first embodiment is preferably data that is sensitive to the spectrum space.

[0039] Step S102; Next, the processing unit 12 inputs the m pieces of time series data group X to a gated recurrent unit (GRU) 21 (see FIG. 4). The GRU 21 inputs the time series data x i The time length l of the time series data x is divided by a predetermined division number S. i 'For each, the partial time series data x i In other words, when the GRU 21 receives m time-series data sets X, it outputs a feature vector set φ slice ∈R m×S×K is output to the K axis. The total number of feature vectors to be output to the K axis can be set in advance in the GRU 21. When the total number of feature vectors is set to a multiple number, multiple feature vectors are output to each of the (m×S) K axes.

[0040] Step S103; Next, the processing unit 12 calculates the feature vector group φ slice ∈R m×S×K The axis order of is converted from "m-axis, S-axis, K-axis" to "K-axis, m-axis, S-axis", and the feature vector u of each K-axis is N (N=m×S) are encapsulated (see Figure 5). These capsules are called primary capsules. This allows the feature vector u N Primary capsule group Ω, which has multiple primary capsules ofprimary {u1,u2,…,u N} is obtained. The conversion of the axis order is a preprocessing step for enabling matrix multiplication to be performed in the graph Fourier transform in step S105.

[0041] Step S104; Next, the processing unit 12 executes graph modeling. Graph modeling is a process for determining weights between primary capsules. That is, the primary capsule set Ω primary {u1,u2,…,u N} is a process for generating a weighting matrix to be applied to

[0042] In graph modeling, a primary capsule is modeled as a graph G(V,E) with a node N(=|V|) and the degree of connection (weight) between adjacent primary capsules as an edge E. As shown in Figure 6, the horizontal edge E1 represents the temporal relationship between primary capsules generated from time series data of one sensor. The vertical edge E2 represents the spatial relationship between primary capsules at the same time generated from time series data of different sensors. The diagonal edge E3 represents both the spatial relationship and the temporal relationship.

[0043] There are two ways to generate a weight matrix: manually setting the weights (attention) between nodes, and learning the weights by machine learning. When manually setting the weight matrix A i,j ∈R N×N is calculated using equation (1).

[0044]

number

[0045] "t i -t j " is the difference between adjacent timestamps in the time axis direction between nodes. i -s j" is the spatial distance between the sensors in the spatial axis direction between the nodes. When manually setting, the difference between the adjacent timestamps and the spatial distance between the sensors are set in advance by a human being. In this regard, in Non-Patent Document 2, as can be seen from equation (1) in Non-Patent Document 2, a weighting matrix is ​​generated based only on the spatial axis. On the other hand, in this embodiment, as shown in the above equation (1), a weighting matrix is ​​generated based on both the spatial axis and the time axis, so that it can also be applied to time series data having feature elements on the time axis.

[0046] On the other hand, in the case of machine learning, the weights of the weighting matrix are machine-learned. The weights of each edge are set arbitrarily in advance, and the weights are automatically updated during the machine learning process. The processing unit 12 generates two matrices W key , W query Then, for these two matrices, we define the primary capsule group Ω primary Then, by multiplying each of them and then applying the softmax function, we obtain the weight matrix A i,j Calculate.

[0047]

number

[0048] This is the weight matrix A i,j At this timing, the processing unit 12 calculates the sum of each column as shown in equation (3) to generate the weight matrix A i,j Diagonal matrix D i,j ∈R N×N Calculate the following.

[0049]

number

[0050] In addition, the processing unit 12 calculates the feature vectors {u1, u2, ..., u N} is the signal value F{f1,f2,…,f NThis replacement is a process to treat the feature vector of the primary capsule as the signal value of the node. i is the feature vector u on the K axis. i is the same value.

[0051] Step S105; Next, the processing unit 12 performs a graph Fourier transform on the time-space domain of the graph model to the spectral domain. Figure 7 shows a case where each of N nodes has three signal values ​​(the total number of feature vectors on the K axis is three). The length of the signal value represents the scalar size of the feature vector. The method of the graph Fourier transform will be described below.

[0052] First, the processing unit 12 calculates a diagonal matrix D i,j From weight matrix A i,j By subtracting, we obtain the graph Laplacian L∈R N×N Calculate.

[0053]

number

[0054] Next, the processing unit 12 performs spectral decomposition of the graph Laplacian L as shown in equation (5) to obtain a unitary matrix U∈R N×Dtrans Transpose of matrix U T Calculate.

[0055]

number

[0056] Then, the processing unit 12 calculates the transpose matrix U of the unitary matrix U for the signal values ​​F of the N nodes as shown in equation (6). T By multiplying by F, the signal value F∈R in the spectral domain is Dtrans×K Calculate (a hat symbol (^) above F). From now on, the hat symbol will be written as simply "(^)".

[0057]

number

[0058] The signal value F(^) after the graph Fourier transform is expressed as "D bases u1 to u Dtrans ×K". In other words, K feature vectors (hereinafter, for ease of understanding, they may be called "dimensions") are output for each basis u. The total number of K is the same as the total number of signal values ​​of each node before the transformation (total number of feature vectors on the K axis). Note that there is no positional correspondence between signal values ​​before and after the graph Fourier transform.

[0059] In this manner, in this embodiment, the signal value F of each node in the time-space domain is graph Fourier transformed to a signal value F(^) in the spectral domain, so that the principle of graph Fourier transform makes it possible to reliably separate the characteristic elements of the signal values ​​between multiple nodes.

[0060] That is, in steps S104 and S105, graph modeling is performed to temporarily assign weights to the nodes as connections. However, since the characteristic elements of each node should differ from node to node, a graph Fourier transform is performed, that is, the time space is converted into a spectral space, and the conversion is made into a frequency axis rather than a spatial axis or a time axis, thereby eliminating the influence between the nodes.

[0061] Step S106; Next, the processing unit 12 performs filtering processing to extract only the target signal value from the K signal values ​​F(^) included in each base u in the spectral domain (see FIG. 8). Specifically, as shown in equation (7), a predetermined filter function g θ By multiplying (∧), the signal value after filtering y(^)∈R Dtrans×K Calculate.

[0062]

number

[0063] In step S106, since it may be unclear which of the K dimensions contained in each basis u in the spectral domain are useful for the final classification task, a filtering process is performed to remove unimportant noise signals so that only the desired dimensions remain.

[0064] Step S107; Next, the processing unit 12 performs preprocessing for performing the graph attention calculation described later, by filtering the signal value y(^)∈R Dtrans×K Let y(^)∈R be the signal value. K×Dtrans (See Figure 9). Specifically, as shown in Figure 10, by classifying the K dimensions included in each basis u into dimensions at the same position, "D basis u × K dimensions" is converted into "K × D dimensions." The unit of this classification is the capsule group Ω(^){z1,z2,…,z K}, z∈R Dtrans , Ω(^)∈R K×Dtrans Let us assume that.

[0065] The multiple dimensions contained in each capsule are completely separated from each other by the graph Fourier transform that has already been performed. It is possible to determine which capsules are important based on the magnitude of the scalars of the dimensions contained in the capsule, and these capsules become the basis for determining the attention of the graph.

[0066] Next, the processing unit 12 calculates the graph attention. Specifically, as shown in equation (8), the graph attention Att is calculated by multiplying the capsule group Ω(^) in the spectral domain by a learnable parameter W. The parameter W is R Dtrans×M where M is the total number of classification tasks.

[0067]

number

[0068] Then, the processing unit 12 calculates the output vector v based on the graph attention Att. Specifically, first, as shown in equation (9), T By multiplying by the capsule group Ω(^), R Dtrans×M The classification result S{s1,s2,…,s m}(s i ∈R Dtrans ) is calculated. The classification result S is calculated as "M × D trans " The matrix is ​​of the form

[0069]

number

[0070] Then, for each classification result s i The output vector v is calculated by applying the squash function in equation (10) to each of the vectors v. The capsule of this output vector is called the high-level capsule.

[0071]

number

[0072] Step S108; Finally, the output unit 13 outputs the output vector v of the high-level capsule.

[0073] To summarize the above, in the first embodiment, a primary capsule is generated by applying GRU processing, division processing, and conversion processing to complex time-series data obtained from multiple sensors, making it possible to store multiple feature elements for each dimension and to realize a learning model that can disentangle each feature element in the time-series data for each feature element.

[0074] In addition, in the first embodiment, for time series data that is sensitive to the spectral space, the time-space relationships are simultaneously graph modeled, and then the graph Fourier transform is applied to calculate the attention of the graph in the spectral domain, thereby separating each feature element in the time series data into individual feature elements, thereby making it possible to realize a learning process that can effectively disentangle the feature elements.

[0075] According to the first embodiment, the information processing device 1 includes an input unit 11 that inputs a plurality of time series data measured by a plurality of sensors at different positions, a processing unit 12 that divides the plurality of time series data into a plurality of partial time series data at a predetermined time interval, generates a plurality of primary capsules including each feature vector of the plurality of partial time series data related to the plurality of time series data, performs graph modeling to generate a weight matrix that expresses the connection relationship between the plurality of primary capsules with weights according to the distance between the plurality of sensors and the predetermined time interval, and performs graph Fourier transform on each feature vector of the plurality of primary capsules based on the weight matrix, so that the method can be applied to multidimensional time series data having feature elements on the time axis, and each feature element can be completely separated from the other feature elements. As a result, a technology that can improve the classification accuracy of each feature element included in multidimensional time series data having feature elements in time and space can be provided.

[0076] In the first embodiment, the primary capsule is generated by performing GRU processing, division processing, and conversion processing on the time series data, but this is an example of a method for generating the primary capsule. Other processing may be used as long as it is possible to separate the spatiotemporal feature elements. For example, convolution processing or the like may be used.

[0077] [Second embodiment] In the first embodiment, the spatiotemporal feature elements are separated by the graph Fourier transform. This is suitable for time series data in which it is difficult for humans to separate the feature elements. On the other hand, depending on the type of time series data, the feature elements may be easily separated. Therefore, in the second embodiment, a method is described in which the feature elements in the time series data are encapsulated separately on the spatial axis and the time axis so that the method can be easily applied to the latter type of time series data.

[0078] The functional block configuration of the information processing device 1 according to the second embodiment includes an input unit 11, a processing unit 12, an output unit 13, and a storage unit 14, similar to the first embodiment shown in Fig. 1. The input unit 11, the output unit 13, and the storage unit 14 have the same functions as those described in the first embodiment.

[0079] The processing unit 12 has a function of using a plurality of time series data to separately generate a plurality of primary capsules each including each feature vector for each time series data, and a plurality of primary capsules each including each feature vector for each predetermined time interval.

[0080] In addition to generating the two primary capsules, the processing unit 12 has a function of generating feature vectors by extracting overall features contained in the time series data for each partial region using the time series data, and generating primary capsules each including each feature vector for each predetermined time interval.

[0081] The processing unit 12 has a function of generating three digit capsules by performing attention routing on each of the three primary capsules, and inferring a task based on the magnitude of the feature vectors included in the three digit capsules.

[0082] Fig. 11 is a diagram showing a processing flow of the information processing device 1 according to the second embodiment. Fig. 12 is a diagram showing a processing image of the information processing device 1 according to the second embodiment.

[0083] Step S201; The input unit 11 receives c pieces of time series data X{x1, x2, ..., x c}(x i ∈R l ) is input. c is the total number of time series data. c is also called the channel. R is the real number space. l is the time series data x i where l is the time length. l is also the total number of time stamps (measurement times). The time series data is, for example, position data of the (x, y, z) coordinates of a hand, elbow, etc., which changes over time. It is preferable that the time series data handled in the second embodiment is time series data other than "spectral space sensitive data".

[0084] Step S202; Steps S202 to S204 constitute a first branch process for acquiring the overall characteristics of the time-series data group X. In this embodiment, the overall characteristics are acquired by performing a convolution process. This process will be described below.

[0085] First, the processing unit 12 inputs c time-series data groups X to the first convolutional layer 31, and inputs the output of the first convolutional layer to the second convolutional layer 32.

[0086] The first convolution layer 31 performs convolution processing on a set of c time series data X using a filter with a kernel size k1 and a target channel number c1. c×l Let X1∈R be the time series data set. c1×(l-k1+1) Convert the data into a time series data set.

[0087] The second convolutional layer 32 uses a filter with a kernel size k2 and a number of target channels c2 to c1×(l-k1+1) Convolution processing is performed on the time series data set of X1∈R. c1×(l-k1+1) Let X2∈R be the time series data set. c2×(l―k1―k2+2) Convert the data into a time series data set.

[0088] The first convolutional layer 31 and the second convolutional layer 32 perform various processes, such as 1D batch normalization, application of a ReLU function, dropout at a rate of 0.3, and application of a squeeze and excitation (SE) block.

[0089] Thereafter, the processing unit 12 inputs the output of the second convolutional layer 32 to the third convolutional layer 33. The third convolutional layer 33 uses a filter with a kernel size k3 and a target channel number c3 to c2×(l-k1-k2+2) This performs convolution processing on the time series data set of X2∈R c1×(l-k1-k2+2) Let X3∈R be the time series data set. c3×l1 The time series data is converted to (l1=l-k1-k2-k3+3). In the third convolutional layer, the SE block is not applied.

[0090] Step S203; Next, the processing unit 12 calculates a learnable matrix W1∈R for the time-series data group X3 as shown in equation (11). c3×l1 By calculating l1 A first primary capsule group Ω1 having a plurality of primary capsules primary {u1,u2,…,u l1 Then, the processing unit 12 calculates the Ω1 primary Ω1 primary ∈R 1×c3×l1 Transforms into.

[0091]

number

[0092] Step S204; Next, the processing unit 12 calculates the first primary capsule group Ω1 as shown in equation (12). primary For Routing Attention A1∈R cls×1×l1 By calculating φ1 digit ∈R cls×c3×l1Calculate the following: cls is the total number of classification classes.

[0093]

number

[0094] Thereafter, the processing unit 12 digit By summing up the last dimension of digit ∈R cls×c3 Calculate.

[0095] Steps S205 to S208 (Overview); After step S201, steps S205 to S208 are executed in parallel with steps S202 to S204. Steps S205 to S208 are a second branch process for dividing the time series data group X along the spatial axis and acquiring features for each channel. In this embodiment, the features for each channel are acquired using a GRU, as in the first embodiment. The process will be described below.

[0096] Step S205; First, the processing unit 12 calculates a set of c time series data X∈R l In the spatial axis direction, c channel groups X set {x1,x2,…,x c} to get

[0097] Step S206; Next, the processing unit 12 selects the channel group X set Each GRU 34 inputs the time series data of each channel included in the . Each GRU 34 shares the same parameters, and after inputting the time series data of each channel, it converts one time series data into c gru Output feature vectors.

[0098] Then, the processing unit 12 calculates the hidden state φ′∈R for each channel output from each GRU 34. cgru Using φ2 primary {φ'1,φ'2,…,φ' c}(φ2primary ∈R cgru×c ) to calculate

[0099] Step S207; Next, the processing unit 12 calculates φ2 primary A learnable matrix W2∈R cgru×c By calculating the feature vector w c A second primary capsule group Ω2 having multiple primary capsules of primary {w1,w2,…,w c The length of the signal value included in each capsule of the primary capsule group (spatial axis) shown in FIG. 12 represents the feature amount in one channel. Then, the processing unit 12 calculates the Ω2 primary Ω2 primary ∈R 1×cgru×c Transforms into.

[0100]

number

[0101] Step S208; Next, the processing unit 12 calculates the second primary capsule set Ω2 as shown in equation (14). primary For Routing Attention A2∈R cls×1×c By calculating φ2 digit ∈R cls×cgru×c Calculate the following: cls is the total number of classification classes.

[0102]

number

[0103] Thereafter, the processing unit 12 digit By summing up the last dimension of digit ∈R cls×cgru Calculate.

[0104] Steps S209 to S212 (Overview); After step S201, steps S209 to S212 are executed in parallel with steps S202 to S204. Steps S209 to S212 are a third branch process that divides the time series data group X on the time axis and acquires features for each timestamp. In this embodiment, like the first embodiment, a GRU is used to acquire features for each timestamp. The process will be described below.

[0105] Step S209; First, the processing unit 12 calculates a set of c time series data X∈R l The GRU 35 inputs a set of c time series data X, and then outputs c gru Output feature vectors.

[0106] Step S210; Next, the processing unit 12 calculates the hidden state ω∈R for each timestamp output from the GRU 35. cgru Using φ3 primary {ω1,ω2,…,ω l}(φ3 primary ∈R cgru×l ) to calculate

[0107] Step S211; Next, the processing unit 12 calculates φ3 primary A learnable matrix W3∈R cgru×l By calculating l A third group of primary capsules Ω3, which has multiple primary capsules of primary {v1,v2,…,v l The length of the signal value included in each capsule of the primary capsule group (time axis) shown in FIG. 12 represents the feature amount at one time stamp. Then, the processing unit 12 calculates the Ω3 primary Ω3 primary ∈R 1×cgru×l Transforms into.

[0108]

number

[0109] Step S212; Next, the processing unit 12 calculates the third primary capsule group Ω3 as shown in equation (16). primary For Routing Attention A3∈R cls×1×l By calculating φ2 digit ∈R cls×cgru×l Calculate the following: cls is the total number of classification classes.

[0110]

number

[0111] Thereafter, the processing unit 12 digit By summing up the last dimension of digit ∈R cls×cgru Calculate.

[0112] Step S213; Next, the processing unit 12 generates the first to third digit capsule groups Ω k digit After applying the batch normalization process and the Rel function to each of (k∈{1, 2, 3}), the norm V k {vote1 k , … vote cls k} Calculate the nome vote j k is an index indicating the confidence of the j-th classification class contained in the k-th digit capsule group.

[0113] Then, the processing unit 12 calculates the norm V for the first to third digit capsule groups as shown in equation (17). k For (k∈{1, 2, 3}), we calculate the final voting result by multiplying and summing the learnable attention weight set (a1, a2, a3).

[0114]

number

[0115] That is, the correct answer for each digit capsule group can be determined by the length of each feature vector contained in the digit capsule group, but since the correct answer may differ for the three digit capsule groups, in step S213, the correct answer is selected by voting based on weights. The voting method is weighted, for example, with three scalars as described above. A large weight is set for what is considered to be a correct answer, and a small weight is set for what is considered to be an incorrect answer. The weights may be set by a human or may be learned weights.

[0116] Step S214; Finally, the output unit 13 outputs the voting result as the inference result.

[0117] To summarize the above, in the second embodiment, for time series data other than "time series data sensitive to spectral space", a spatial axis and a primary capsule group not separated on the spatial axis, a primary capsule group separated on the spatial axis, and a primary capsule group separated on the time axis are generated, and a routing operation is applied to each of them. The three generated digit capsules are integrated by a learnable voting mechanism, thereby making it possible to improve classification accuracy and improve the interpretability of the learning process through branching analysis, etc.

[0118] According to the second embodiment, the information processing device 1 includes an input unit 11 that inputs a plurality of time series data measured by a plurality of sensors at different positions, and a processing unit 12 that uses the plurality of time series data to generate a plurality of primary capsules each including a feature vector for each time series data and a plurality of primary capsules each including a feature vector for each predetermined time interval, so that the second embodiment can be applied to multidimensional time series data having feature elements on the time axis, and each feature element can be completely separated from the other feature elements. As a result, a technology can be provided that can improve the classification accuracy of each feature element included in multidimensional time series data having feature elements in time and space.

[0119] [others] The present invention is not limited to the above embodiment. Various modifications of the present invention are possible within the scope of the present invention. The first and second embodiments may be combined.

[0120] The information processing device 1 of the first and second embodiments described above can be realized, for example, by using a general-purpose computer system including a CPU 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 13. The memory 902 and the storage 903 are storage devices. In the computer system, the CPU 901 executes a predetermined program loaded on the memory 902, thereby realizing each function of the information processing device 1.

[0121] The information processing device 1 may be implemented by one computer. The information processing device 1 may be implemented by multiple computers. The information processing device 1 may be a virtual machine implemented in a computer. The program for the information processing device 1 may be stored in a computer-readable recording medium such as an HDD, SSD, USB memory, CD, or DVD. The program for the information processing device 1 may also be distributed via a communication network. [Explanation of symbols]

[0122] 1: Information processing device 11: Input section 12: Processing section 13: Output section 14: Storage part

Claims

1. an input unit for inputting a plurality of time series data measured by a plurality of sensors at different positions; a processing unit that divides the plurality of time series data into a plurality of partial time series data at a predetermined time interval, generates a plurality of primary capsules each including a feature vector of the plurality of partial time series data related to the plurality of time series data, performs graph modeling to generate a weight matrix that expresses a connection relationship between the plurality of primary capsules with a weight according to the distance between the plurality of sensors and the predetermined time interval, and performs a graph Fourier transform on each feature vector of the plurality of primary capsules based on the weight matrix; An information processing device comprising:

2. The processing unit includes: The information processing device according to claim 1, further comprising: generating a plurality of capsules in the spectral domain by classifying a plurality of signal values ​​contained in a plurality of bases obtained by the graph Fourier transform according to signal values ​​at the same position within the base; and calculating attention for classification classes based on the magnitude of the signal values ​​contained in the plurality of capsules.

3. An information processing method performed by an information processing device, A step of inputting a plurality of time series data measured by a plurality of sensors at different positions; a step of dividing the plurality of time series data into a plurality of partial time series data at a predetermined time interval, generating a plurality of primary capsules each including a feature vector of the plurality of partial time series data related to the plurality of time series data, performing graph modeling to generate a weight matrix that expresses a connection relationship between the plurality of primary capsules with a weight according to the distance between the plurality of sensors and the predetermined time interval, and performing a graph Fourier transform on each feature vector of the plurality of primary capsules based on the weight matrix; An information processing method for performing the above.

4. An information processing program that causes a computer to function as the information processing device according to claim 1 or 2.

Images