Time-series data processing techniques for statistical machine learning

Abstract

To enable an efficient process for even time-series data with a small data amount by a small amount of calculation or a lightweight statistical machine learning model.SOLUTION: An information processing device 10 includes: a division unit 14 that divides time-series data into a plurality of blocks including feature amounts; a shuffle unit 15 that at least partially rearranges a time series order of the plurality of blocks; and an output unit 16 that outputs feature amount series data obtained by rearranging the blocks to a statistical machine learning model 17.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing technique for time series data suitable for statistical machine learning. Specifically, the present invention relates to an information processing device, a computer program, and an information processing method for processing time series data. The present invention also relates to a method for producing feature sequence data for statistical machine learning. Furthermore, the present invention relates to a neural network structure including a processing layer for time series data. [Background technology]

[0002] When dealing with time series data in statistical machine learning, the specific time resolution (i.e., the time interval for capturing the physical quantities of the phenomenon and their changes) is important for analyzing the phenomenon of interest. For example, while short-term fluctuations play an important role in some classification problems, long-term fluctuations play an important role in some regression problems.

[0003] A method for processing time series data typically has the following structure. First, the time series data is divided into a number of short blocks, and features are extracted from the blocks by moving average, short-time Fourier transform, feature extraction based on domain knowledge, neural network, or the like. The blocks here have a time resolution sufficient to extract desired features. Next, the feature sequence for each block is processed by a method for calculating statistics, or by a statistical method for handling time series data (recurrent neural network, hidden Markov model, etc.). For example, in the former case, Patent Document 1 discloses a technique for dividing the frequency of the time series data into blocks for each predetermined frequency width, and extracting features based on the blocked frequencies. In the latter case, Patent Document 2 discloses a technique for recognizing information input in a time series using a neural network. However, in such conventional methods, the order of the feature sequence is either completely ignored and reduced to statistics, or is completely preserved and handled. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication WO2015 / 071925 Brochure [Patent Document 2] JP 2019-159654 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, the method of dividing time series data into a certain number of blocks and calculating statistics for feature sequences for each block has the problem that it is not possible to consider fluctuations on a block-by-block basis. In addition, this method also has the problem that it can only handle simple information because it reduces information to a small number of statistics such as the mean and variance. Furthermore, long series data are required both during learning and inference to ensure the reliability of the statistics.

[0006] In addition, when using statistical methods such as recurrent neural networks to process time series data, fluctuation patterns on time scales that are unnecessary for the target machine learning task are also retained, making it difficult to extract only the necessary information in advance from the target time series data. Furthermore, in time series data, essential information may be localized in time, but when processing time series data sequentially along the time axis using statistical methods, information often cannot be accumulated and essential information is lost. Therefore, the efficiency of information processing becomes very low, resulting in low accuracy, or a huge amount of data and calculations are required during learning to achieve high accuracy.

[0007] Therefore, a main object of the present invention is to enable efficient processing of even small amounts of time-series data using a small amount of calculation or a lightweight statistical machine learning model. [Means for solving the problem]

[0008] The inventors of the present invention have studied intensively the means for solving the problems of the above-mentioned conventional technology, and as a result, have decided to divide the time series data to be processed into a plurality of blocks at a time resolution at which the feature amount is not lost, rearrange the time series order of each block, and then output it to a statistical machine learning model. That is, in time series data, essential information is often localized or unevenly distributed at a time resolution specific to the phenomenon of interest or the machine learning task. In addition, a large amount of unnecessary information may be embedded in the time series data. For this reason, when processing time series data, it is a major issue to determine from which time range to extract information. In the present invention, as a solution to this issue, a method of rearranging the time series order of the time series data is proposed. In appropriately rearranged time series data, unnecessary time resolution information is removed, and only information of the time resolution essential to the phenomenon of interest is retained. In addition, rearranging the time series data homogenizes the amount of essential information per time. In addition, since various sorting methods have the same effect as long as essential information is preserved, it is possible to increase the substantial number of data, i.e., expand the data, by implementing multiple sorting methods. Therefore, by using such a method, it is possible to efficiently process time series data using a statistical machine learning method that handles time series data, such as a neural network with a recurrent structure. Specifically, the present invention has the following configuration or steps.

[0009] A first aspect of the present invention relates to an information processing device for performing information processing of time series data with high accuracy and efficiency. "Time series data" is information in which a physical quantity of a certain phenomenon or its change amount is represented by values ​​that are continuous over time. A representative example of "time series data" handled in the present invention is sound data. The sound data is preferably a sound with a frequency in the human audible range such as voice, music, or environmental sound, but may be other sounds in the inaudible range such as ultrasonic waves or low frequencies. In addition, the time series data that can be handled in the present invention is not limited to sound data, and various time series data such as video data, still image data, character string data, sales data, stock price data, and weather data can be handled.

[0010] The information processing device according to the present invention includes a division unit, a shuffle unit, and an output unit.

[0011] The dividing unit divides the time series data into a plurality of blocks each including a feature. For example, sound data may be divided into blocks at regular time intervals, or features such as phonemes and words may be extracted from the sound data and then divided into blocks. Video data may be divided into blocks for a predetermined number of frames, for example. Still image data may be divided into blocks for local pixels or horizontal lines, for example, by regarding the spatial direction as the time direction. Character string data may be divided into blocks for each character, word, or utterance, for example, by regarding the character string direction as the time direction. Other time series data may be divided into blocks for each second, minute, hour, day, month, year, or the like.

[0012] The shuffle unit at least partially rearranges the time series order of the multiple blocks obtained by the division unit. Typical examples of rearrangement include a process of replacing the order of a certain object (such as a block) with another object, and a process of inserting a certain object into a different order. There are various possible modes of rearrangement by the shuffle unit, and examples of these will be described in detail later.

[0013] The output unit outputs the feature sequence data obtained by the sorting to the statistical machine learning model. The statistical machine learning model is not particularly limited and can adopt a known model. For example, the statistical machine learning model can be a supervised learning model such as a deep learning model, an unsupervised learning model such as a reservoir computation model, or a model that does not learn at all, and the structure and learning method are not important. In addition, the term statistical machine learning model includes both a model in the learning phase (i.e., a model before learning) and a model in the inference phase (i.e., a model that has been learned). In this specification, the data before the sorting process is basically referred to as "time series data" and the data after the sorting process is basically referred to as "feature sequence data".

[0014] In the information processing device according to the present invention, the dividing unit may further divide the time series data into a plurality of upper blocks each including a plurality of blocks. The number of blocks included in the upper block is arbitrary, but may be, for example, about 2 to 10. In this case, the shuffling unit may rearrange the chronological order of the plurality of blocks in the upper block, or rearrange the chronological order of the upper block.

[0015] More specifically, the shuffle unit can at least partially rearrange the chronological order of the upper block while maintaining the chronological order of the blocks in the upper block (LONG-TERM SHUFFLE). Also, the shuffle unit can at least partially rearrange the chronological order of the blocks in the upper block while maintaining the chronological order of the upper block (SHORT-TERM SHUFFLE). As to which of these rearrangement methods to adopt, an appropriate method can be selected in consideration of the properties of the time series data, the phenomenon of interest, and the machine learning task.

[0016] In the information processing device according to the present invention, the shuffle unit may determine whether or not a certain block is to be rearranged and another block to be replaced with the certain block based on a predetermined probability distribution (PROBABILISTIC SHUFFLE). In this case, it is not essential to classify a plurality of blocks into higher order blocks. The probability distribution may be a continuous probability distribution such as a normal distribution, or a discrete probability distribution such as a binomial distribution or a geometric distribution.

[0017] In the information processing device according to the present invention, the dividing unit may hierarchically organize a plurality of blocks included in the time series data. For example, the time series data is hierarchically organized such that a predetermined number of blocks are grouped into a first upper block and a predetermined number of the first upper blocks are grouped into a second upper block. The shuffling unit determines whether or not to rearrange the chronological order of the blocks, or a rearrangement method (LONG-TERM SHUFFLE, SHORT-TERM SHUFFLE, PROBABILISTIC SHUFFLE, etc.) for each layer. Specifically, it can determine whether or not to perform rearrangement processing for each layer. Also, different rearrangement methods may be executed for each layer, or the same rearrangement method may be executed in some cases.

[0018] In the information processing device according to the present invention, the output unit may output the original time series data, in which the time series order of the features has not been rearranged, to the statistical machine learning model together with the feature series data obtained by rearrangement. The output unit may merge the feature series data obtained by rearrangement with the original time series data and then output the merged data to the statistical machine learning model. This makes it possible to handle both the case where information of a certain time resolution is retained and the case where information is randomized in parallel. There are various options for the stage at which the branched paths are merged. For example, the information of the branched paths may be merged and then output to a machine learning model that handles a single time series data. In addition, at the stage of outputting information to the machine learning model, information may be transferred independently for each branched path. Note that the information may be merged inside the machine learning model. In addition, a machine learning model is prepared for each branched path, and information is output to each machine learning model. In this case, the information may be merged at the final stage of the machine learning model.

[0019] A second aspect of the present invention relates to a computer program for processing time series data. The program according to the present invention causes a general-purpose computer to function as the information processing according to the first aspect described above (i.e., an information processing device including a division unit, a shuffle unit, and an output unit). The program according to the present invention may be recorded on a recording medium such as a CD-ROM, or may be downloadable via an information communication line such as the Internet.

[0020] A third aspect of the present invention relates to an information processing method by a computer. In the information processing method according to the present invention, a computer executes a dividing step, a rearrangement step, and an output step. The dividing step is a step of dividing time series data into a plurality of blocks including features. The rearrangement step is a step of at least partially rearranging the time series order of the plurality of blocks. The output step is a step of outputting the feature series data obtained by the rearrangement to a statistical machine learning model.

[0021] A fourth aspect of the present invention relates to a method for producing feature sequence data for statistical machine learning by a computer. In the method for producing feature sequence data according to the present invention, a computer executes a division step and a rearrangement step. The division step is a step of dividing time series data into a plurality of blocks including features. The rearrangement step is a step of at least partially rearranging the time series order of the plurality of blocks. The feature sequence data obtained in the rearrangement step is used as teacher data, for example, when learning a statistical machine learning model.

[0022] A fifth aspect of the present invention relates to a neural network structure. The neural network structure according to the present invention includes at least a shuffle layer and a learning layer. The shuffle layer divides time series data into a plurality of blocks including features, and at least partially rearranges the time series order of the plurality of blocks to obtain feature sequence data. The learning layer performs statistical machine learning based on the feature sequence data obtained by the shuffle layer. In this neural network structure, the rearrangement process by the shuffle layer can be performed not only during learning using the time series data, but also during testing (inference) such as pattern analysis of the time series data. Effect of the Invention

[0023] As configured above, in the information processing device according to the present invention, the order of target time series data is randomized in other time scales while maintaining the fluctuation pattern in a certain time scale of interest. This provides the following specific effects [1] to [3]. Due to these effects, the present invention makes it possible to efficiently process even time series data with a small amount of data by using a small amount of calculation or a lightweight statistical machine learning model.

[0024] [1] Efficiency in selecting information In this invention, information about time series patterns on time scales that are not of interest is removed from the feature sequence through sorting. This eliminates the need to remove unnecessary information in the downstream statistical learning model that handles time series patterns. This makes it possible to process time series data efficiently with a lightweight model, a small amount of calculation, and a small amount of data.

[0025] [2] Homogenizing the amount of information Time series data does not necessarily contain information that is useful for the target machine learning task in a uniform manner in the time direction. In this regard, in the present invention, the distribution of useful information in the time direction is homogenized by a sorting process. Therefore, a statistical learning model that handles time series patterns does not need to keep long-term information when sequentially handling time series data. Therefore, it becomes possible to efficiently process time series data with a lightweight model, a small amount of calculation, and a small amount of data.

[0026] [3] Data augmentation The order of the blocks that make up the time series data is rearranged randomly. Therefore, even if the original time series data is the same, different feature sequence data can be output by rearranging it in a different way. This makes it possible to generate a large amount of feature sequence data useful for training statistical machine learning models from the original time series data. In other words, data expansion is possible. Therefore, even if only a small amount of time series data is available, it becomes possible to train a model that handles time series data efficiently. [Brief description of the drawings]

[0027] [Figure 1] FIG. 1 is a block diagram illustrating an example of a functional configuration of an information processing device. [Diagram 2] FIG. 2 shows a schematic diagram of a process for dividing time series data into multiple blocks. [Diagram 3] FIG. 3 shows a schematic diagram of the rearrangement process (LONG-TERM SHUFFLE) of high-order blocks. [Figure 4]FIG. 4 shows a schematic diagram of a process of rearranging blocks in a higher-order block (SHORT-TERM SHUFFLE). [Diagram 5] FIG. 5 shows an application example of the sorting process. [Figure 6] FIG. 6 shows a schematic diagram of the hierarchical organization of blocks of time-series data. [Figure 7] FIG. 7 shows a schematic diagram of a sorting process based on a probability distribution (PROBABILISTIC SHUFFLE). [Figure 8] FIG. 8 shows a schematic example of the sorting process. [Figure 9] FIG. 9 is a block diagram showing an example of a functional configuration in the case where a processing sequence is branched into time-series data with the original order maintained and rearranged feature amount series data. [Figure 10] FIG. 19 is a block diagram showing an example of a neural network structure. [Figure 11] FIG. 11 is a block diagram showing an example of a neural network structure including path branching. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiments described below, and includes appropriate modifications of the embodiments described below within the scope obvious to those skilled in the art.

[0029] FIG. 1 shows functional blocks of an information processing device 10 according to an embodiment of the present invention. The information processing device 10 is configured by a general-purpose computer, and performs calculations on information inputted through an input device by a control and arithmetic device such as a processor according to a predetermined computer program stored in a storage device such as a storage or memory, and outputs the calculation results through an output device. As shown in FIG. 1, the information processing device 10 includes, as functional blocks, an input unit 11, a preprocessing unit 12, a feature extraction unit 13, a division unit 14, a shuffle unit 15, an output unit 16, a statistical machine learning model 17, and a trained model 18. These functional blocks 11 to 18 are realized by hardware such as the input device, the storage device, the control and arithmetic device, and the output device described above.

[0030] The information processing device 10 according to the present embodiment relates to a neural network-based information processing device capable of processing time-series data and performing learning, inference, or classification. For example, sound (voice, music, etc.), text, video, still images, etc. can be handled as time-series data. In the embodiment described below, it is assumed that sound data is mainly processed as an example of time-series data.

[0031] The input unit 11 accepts input of time series data to be processed. The input unit 11 may read out time series data stored in an internal storage or an external storage medium of the information processing device 10, or may receive time series data via a communication line such as the Internet. The input unit 11 may also accept input of sound data as time series data from a sound collection device such as a microphone.

[0032] The pre-processing unit 12 performs a predetermined pre-processing depending on the type of time-series data and the phenomenon or physical quantity of interest. For example, when the input time-series data is sound data (acoustic signal), the pre-processing unit 12 can perform any processing such as amplifying the sound data, extracting and separating voice components, removing noise, and filtering a predetermined frequency.

[0033] The feature extraction unit 13 extracts features for learning, inference, or identification from the time series data. When the time series data is sound data, the feature extraction unit 13 extracts sound features from the sound data. An example of the feature extraction unit 13 is a deep neural network (DNN). The DNN may be a convolutional neural network (CNN) having an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer, or a recurrent neural network (RNN) having an input layer, a hidden layer, and an output layer and recurrently generating values ​​of the hidden layer. In addition, it is also possible to adopt a known neural network structure such as a long short-term memory (LSTM) having a data storage unit in which data from a long time ago is associated and recorded.

[0034] The division unit 14 divides the time series data into a plurality of blocks in the time axis direction. The division unit 14 performs division so that each block has a time length that does not lose the feature amount extracted by the feature extraction unit 13. The time length of each block can be set arbitrarily according to the phenomenon or physical quantity of interest. For example, when focusing on the phonemes of the sound data, the blocks can be set to a length corresponding to the phonemes, and when focusing on single sounds or words, the blocks can be set to a length corresponding to each of them. Figures 2(a) and (b) show a schematic diagram of the process of dividing the time series data into a plurality of blocks having feature amounts.

[0035] The shuffle unit 15 rearranges the time-series order of blocks constituting the time-series data. Typical rearrangement processes include a process (Swap) of swapping the order of a certain object (such as a block) with another object as shown in FIG. 2(c1), and a process (Insertion) of inserting a certain object into a different order as shown in FIG. 2(c2). In the case of a swap process, only the order of the two target blocks is swapped, and the order of the other blocks remains unchanged. In the case of a swap process, a certain object (such as a block) may be swapped two or more times, or an object that has already been swapped may be stored so that the same object is not swapped two or more times. On the other hand, in the case of an insertion process, the order of the time series of blocks other than the block that is the object of the process will also change. Similarly, in the case of an insertion process, a certain object (such as a block) may be inserted into a different order two or more times, or an object that has already been inserted may be stored so that the same object is not inserted two or more times.

[0036] Next, various methods of rearrangement processing by the shuffle unit 15 will be described in more detail with reference to FIG. 3 to FIG.

[0037] FIG. 3 shows a process (LONG-TERM SHUFFLE) for defining a plurality of upper blocks including a plurality of blocks and rearranging the upper blocks. As shown in FIG. 3(a), first, the division unit 14 defines an upper block that groups a plurality of blocks together while maintaining the time series order of each block. In each upper block, the order of the plurality of blocks remains the same as the original time series data. In the example shown in FIG. 3, five blocks are included in one upper block. The number of blocks in the upper block can be set arbitrarily, for example, in the range of 2 to 10.

[0038] Next, the shuffle unit 15 rearranges the order of the upper blocks while maintaining the order of the blocks in the upper blocks. The shuffle unit 15 may specify two upper blocks and interchange the order of these two upper blocks as shown in Fig. 3(b1). Alternatively, the shuffle unit 15 may specify one upper block and insert the order of the specified upper block into another order as shown in Fig. 3(b1). Such a rearrangement process is performed on multiple upper blocks constituting the time series data.

[0039] Such rearrangement of the upper blocks means that the fluctuation pattern is preserved on a short time scale (within the upper block) while the fluctuation pattern is randomized on a long time scale. Therefore, this rearrangement method is effective when the fluctuation pattern on the time scale of interest is on a short time scale (within the upper block). Information on the time series pattern on a long time scale that is not of interest is randomized and excluded from the feature sequence.

[0040] Fig. 4 shows a process (SHORT-TERM SHUFFLE) for defining multiple upper blocks each including multiple blocks and rearranging the blocks in the upper blocks. As shown in Fig. 4(a), first, the division unit 14 defines an upper block that groups multiple blocks together while maintaining the chronological order of each block. Up to this point, the process is the same as that in Fig. 3.

[0041] Next, the shuffle unit 15 rearranges the order of the blocks in the upper block while maintaining the order of the upper blocks. As shown in Fig. 4(b1), the shuffle unit 15 may specify two blocks in a certain upper block and swap the order of these two blocks. As shown in Fig. 4(b1), the shuffle unit 15 may also specify one block in a certain upper block and insert the order of the specified block into another order in the same block. This sorting process is performed for each upper block constituting the time series data.

[0042] Such rearrangement of blocks within a higher-level block means that the fluctuation pattern is preserved on a longer time scale, while the fluctuation pattern is randomized on a shorter time scale (within the higher-level block). Therefore, this rearrangement method is effective when the fluctuation pattern on the time scale of interest is a longer time scale. Information on the time series pattern on the shorter time scale (within the higher-level block) that is not of interest is randomized and excluded from the feature sequence.

[0043] FIG. 5 shows an example of a change in the sorting process shown in FIG. 3 and FIG. 4. As shown in FIG. 5(a), the number of blocks contained in a higher-order block may be different for each higher-order block. For example, the number of blocks contained in a higher-order block may be randomly determined from a range of 2 to 10. Also, as shown in FIG. 5(b), it is possible to apply a combination of a process of sorting higher-order blocks (LONG-TERM SHUFFLE) and a process of sorting blocks in a higher-order block (SHORT-TERM SHUFFLE) to certain time-series data. In this case, a process of swapping the order of two objects (blocks or higher-order blocks) (Swap) and a process of inserting an object into another order (Insertion) may be applied in combination.

[0044] FIG. 6 shows an example of hierarchical arrangement of blocks. As shown in FIG. 6(a), first, the division unit 14 defines a first upper block that groups a plurality of blocks together while maintaining the time series order of each block. Furthermore, the division unit 14 determines a second upper block that groups a plurality of first upper blocks together. The number of blocks included in the first upper block and the number of first upper blocks included in the second upper block can be set arbitrarily. In the example shown in FIG. 6, at least two second upper blocks are formed. Therefore, the time series data can be considered to be divided into at least two hierarchical levels. In this way, the second upper block is a block for forming the boundary between hierarchical levels, and the suitability of the sorting process or the method of the sorting process is determined for each hierarchical level (for each second upper block). The number of hierarchical levels (second upper blocks) can be increased to any number as long as the length of the time series pattern allows.

[0045] For example, as shown in FIG. 6(b1), the shuffle unit 15 can rearrange the first upper blocks in a certain second upper block, and rearrange the blocks in the first upper block while maintaining the order of the first upper blocks in another second upper block. In the example of FIG. 6(b1), the blocks in the first upper block are rearranged while maintaining the order of the first upper blocks in two second upper blocks, but the rearrangement method is different between swap and insertion. In this way, the rearrangement method can be changed for each hierarchy (second upper block). Also, although not shown, it is possible to determine whether or not to perform rearrangement for each hierarchy.

[0046] In this way, by forming a hierarchy (second highest block) and maintaining the order of each hierarchy, consistency between the hierarchical layers is maintained. By introducing such a hierarchical structure, it becomes possible to select whether or not to randomize the time series pattern information for each time resolution, and to select the randomization method.

[0047] FIG. 7 shows a method (PROBABILISTIC SHUFFLE) in which each block is rearranged based on a probability distribution without forming a higher-order block. In the example shown in FIG. 7, a normal distribution is used. The shuffle unit 15 first determines whether each block is to be rearranged. Specifically, when the block "09" is selected as a target, the block "09" is determined as a target for rearrangement with a probability of 0.5. This probability can be set arbitrarily. Next, a normal distribution is set with the block "09" that is the target of rearrangement at the center and 0.5 at the apex. Next, another block to be swapped with the block "09" is determined. This other block is determined according to the probability of the normal distribution set as described above. As a result, the block "09" has a high probability of being swapped with a block close to it, and the further away the block is, the lower the probability of being swapped. In the example shown in the figure, a normal distribution is given as the probability distribution, but other continuous probability distributions or discrete probability distributions such as binomial distribution and geometric distribution may be used. In addition, in the illustrated example, a block to replace a certain block is determined based on a probability distribution, but the order in which a certain block is inserted may also be determined based on a probability distribution (Insertion).

[0048] Various methods of rearrangement processing have been described above. Here, regardless of which of the above-mentioned rearrangement processing methods is adopted, the shuffle unit 15 may perform the rearrangement processing randomly every time data is processed, as shown in Fig. 8(a). Specifically, as shown in Fig. 8(a), even if the time-series data input to the shuffle unit 15 is the same data or different data, it is possible to make the order of the blocks after rearrangement change randomly in the first, second, third, etc.

[0049] On the other hand, in consideration of calculation efficiency and program implementation efficiency, the shuffle unit 15 can also perform a rearrangement process for certain time series data (such as a mini-batch in DNN learning) so that the rearranged order is the same every time. In this case, different rearrangements are performed between different time series data.

[0050] In addition, as shown in Fig. 8(b), when dividing time series data into multiple upper blocks and performing a process (SHORT-TERM SHUFFLE) to rearrange the blocks in each upper block, it is of course possible to perform the rearrangement process so that the order of the blocks after rearrangement differs for each upper block. Also, the rearrangement method can be changed for each upper block.

[0051] When forming upper blocks as in the methods shown in Figures 3 and 4, it is necessary to determine the upper block boundary by hardwire. In contrast, in the method using probability distribution as shown in Figure 6, it is possible to control the rearrangement probabilistically without setting an upper block boundary. It is preferable that the probability distribution be one in which the typical time difference and statistical variation can be controlled by parameters, and in particular, a normal distribution is preferable.

[0052] Returning to Fig. 1 to continue the explanation of the functional blocks of the information processing device 10, the output unit 16 outputs the feature amount sequence data obtained by rearranging the time series order by the shuffle unit 15 to a downstream statistical machine learning model 17. The output unit 16 can also store the feature amount sequence data in a storage device.

[0053] The statistical machine learning model 17 is a machine learning model suitable for handling time series data. Examples of such models include, but are not limited to, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), and a hidden Markov model.

[0054] In addition, a reservoir computing model can be used as the statistical machine learning model 17. The reservoir computing model is a type of RNN consisting of three layers, an input layer, a reservoir layer (hidden layer), and an output layer. The connection weights between the input layer and the reservoir layer and in the reservoir layer are not changed from their initial values, and the connection weights are adjusted only in the edges connecting the reservoir layer and the output layer. The reservoir layer has nodes connected by edges without regularity, and has the function of learning input information by unsupervised learning and accumulating the learned information. In addition, algorithms that belong to the category of reservoir computing include ESN (Echo State Network) and LSM (Liquid State Machine), both of which have a low computational burden, can handle time-series data, and can obtain learning results that are comparable to RNNs, etc. The reservoir layer of the ESN is composed of a recurrent network of neurons expressed by a nonlinear function neuron model. Although the connections between neurons and their weights are determined randomly, time series data is learned based on the state of neurons that contain past information.

[0055] The learning results of the statistical machine learning model 17 are stored as a trained model 18. This learning result is used to infer and identify patterns in time series data.

[0056] Next, a structure for branching the processing sequence of time series data with the original order maintained and rearranged feature sequence data will be described with reference to Fig. 9. This method makes it possible to handle both the case of retaining information of a certain time resolution and the case of randomization in parallel.

[0057] In the structure shown in FIG. 9(a), the processing sequence of the time series data is branched into a first path in which the feature is extracted by the feature extraction unit 13, and then processed by the division unit 14 and the shuffle unit 15 to the output unit 16, and a second path in which the processing sequence does not pass through these functional blocks 14 and 15 and reaches the output unit 16. In the first path, the feature series data that has been subjected to the division and rearrangement processing is output to the statistical machine learning model 17 by the first output unit 16a. On the other hand, in the second path, the time series data that maintains the original order is output as it is to the statistical machine learning model 17 by the second output unit 16b. After that, the feature series data and the original time series data are integrated inside the statistical machine learning model 17.

[0058] 9(a), although not shown, it is also possible to prepare separate statistical machine learning models 17 for the first path and the second path, and configure the first output unit 16a and the second output unit 16b to output data to the separate statistical machine learning models 17. In this case, the information may be integrated in the final stage of the machine learning models.

[0059] 9(b), the feature sequence data that has been subjected to the division and rearrangement processing in the first pass and the time series data in the second pass with the original order maintained are integrated by the integration unit 16c included in the output unit 18 before being output to the statistical machine learning model 17. In this way, the information of the branched paths can be merged and then output to the statistical machine learning model 17 that handles a single time series data.

[0060] In addition, as shown in Fig. 6, it is possible to introduce a hierarchical structure into the time resolution to hierarchically organize the time series data, and then branch into a first pass (a processing sequence that divides and rearranges) and a second pass (a processing sequence that maintains the original order) at each level. This makes it possible to simultaneously handle the original time series information and randomized time series information at different time resolutions. These extensions make it possible to handle complex time series patterns, such as when the typical time resolution is unknown in advance or when there is a correlation or dependency between multiple time resolutions. Note that a statistical machine learning model17 will determine which resolution is important and how to effectively combine information from different resolutions.

[0061] Next, with reference to FIG. 10, an embodiment in which the above-mentioned rearrangement process of feature strings of time-series data is applied to a layer of a neural network will be described. The neural network structure 20 shown in FIG. 10 includes a shuffle layer 22 between a first learning layer 21 and a second learning layer 23. The shuffle layer 22 includes a division unit 22a and a shuffle unit 22b. The first learning layer 21 basically has a function of extracting feature quantities of time-series data, and has a function equivalent to that of the feature extraction unit 13 shown in FIG. 1. For this reason, any neural network (having an appropriate input / output data structure) can be used for the first learning layer 21. The division unit 22a and the shuffle unit 22b included in the shuffle layer 22 basically have a function equivalent to that of the division unit 14 and the shuffle unit 15 shown in FIG. 1. The rearrangement process by the division unit and the shuffle unit is as described above. The second learning layer 23 basically corresponds to the statistical machine learning model 17 shown in FIG. 1. For this reason, the second learning layer 23 can use a neural network (with an appropriate input / output data structure) including a recurrent structure or an attention mechanism.

[0062] FIG. 11 shows a neural network structure 30 having branched paths as an application example of FIG. 10. In the example shown in FIG. 11, in one path, feature extraction of time series data is performed in the first learning layer 31, rearrangement processing of the time series data is performed in the first shuffle layer 32, and machine learning based on the rearranged feature series data is performed in the second learning layer 33. In another path, feature extraction of the time series data is performed in the first learning layer 31, and then machine learning of the original time series data is performed in the third learning layer 34, and rearrangement processing is performed on the data output from the third learning layer 34 in the second shuffle layer 35. The data processed separately in the two paths are finally integrated in the integration layer 36.

[0063] Considering the above-mentioned hierarchical structure and path branching together, the neural network structure as a whole has the following structural characteristics. That is, time series data is input. It can also have a hierarchical structure corresponding to the time resolution. It can also use a shuffle layer that rearranges the time series in each layer. It can also branch paths in each layer into a system that uses a time rearrangement layer and a system that does not. It can also have an output layer according to the target machine learning task. Any structure other than the above can be designed.

[0064] In addition, in the neural network structure according to the present invention, supervised learning may be performed using an error backpropagation method, as in deep learning, or unsupervised learning may be performed using reservoir calculation, or the initial value may be fixed to a random value. Also, a different learning method may be adopted for each learning layer. In either case, the shuffle layer does not include a learnable parameter. As an example, a network structure in which a Temporal Convolutional Network (TCN) is extended to alternately stack CNN and shuffle layers is considered. As another example, a network structure in which a deep LSTM is extended to alternately stack LSTM and shuffle layers is considered.

[0065] Next, as an application example of the information processing device or neural network structure according to the present invention, a concrete example of sentence-independent (a condition in which the spoken sentence is different between registration and recognition) speaker identification will be described. Here, an application example of deep learning and reservoir calculation, particularly a shuffle layer to two types of ESN statistical machine learning models (ESN and x-vector), will be described.

[0066] Here, the voice signal is often converted into a spectrogram in the time-frequency domain for processing. It is difficult to identify speakers using frame-by-frame short-time Fourier transforms. However, since it is possible to distinguish even short voices if it is a familiar voice, it is expected that speaker characteristics are embedded even in relatively short speech segments. In fact, a method called x-vector, which currently achieves the highest level of performance, uses a three-layered neural network to embed low-dimensional information from a 15-frame voice segment, and then converts it for each time frame before calculating the average. On the other hand, for speaker identification, long-time scale fluctuation patterns such as phoneme sequences are considered to be unnecessary information.

[0067] Therefore, when applying a shuffle layer to x-vector, the shuffle layer is inserted between the second and third layers of the neural network. When applying a shuffle layer to ESN, the mel-log-spectrogram is used as the feature, the frames of the short-time Fourier transform are used as the block, and the 15-frame speech section is used as the upper block, and LONG-TERM SHUFFLE is used. The spectrogram is transformed by a self-organizing map (separately trained without a teacher) and then handed over to the reservoir layer. Recognition is performed by linear regression of the speaker vector. By introducing the sorting of the time series data in this way, the fluctuation patterns of long time scales are randomized while maintaining the local fluctuations of the feature. In addition, short speech (e.g., 1000 milliseconds) is used for both learning and recognition. The same speech is input 10 times for both speaker registration and speaker recognition. As a result, due to the effects of homogenization of the amount of temporal information, information selection, and data expansion by time sorting, higher speaker recognition accuracy is achieved compared to the case where sorting is not performed. It is believed that using time reordering in this manner allows for efficient processing of speech data, even for very short speech segments.

[0068] The information processing device and neural network structure according to the present invention are applicable to cases where time-series data is handled by statistical machine learning. The present invention is expected to be particularly effective when used in combination with a neural network having a recurrent structure. In particular, as statistical machine learning models, RNNs that perform supervised learning, unsupervised learning, or reservoir calculations that are fixed with random initial values ​​are useful. Furthermore, by using the present invention in combination with reservoir calculations (ESN, LSM, etc.), it is possible to apply the present invention to cases where a small amount of data, a small amount of calculation resources, and low latency are required. Furthermore, it is also possible to implement the present invention in hardware using a neuromorphic chip.

[0069] The first example of a task for which the present invention is effective is the classification of voice data. For example, the present invention can be suitably used in tasks such as speaker recognition, emotion recognition, environmental sound recognition, and attitude recognition. For image data, the spatial direction of a still image can be regarded as, for example, a sequence of pixels in the time direction, which is useful for detecting a global structure that remains even when local pixels are shuffled, recognizing a local texture ignoring the global structure, or combining these hierarchical structures. For video data, the present invention is useful in tasks that handle time-series patterns, such as behavior classification. For natural language processing, the character string direction can be regarded as, for example, a sequence of characters in the time direction, which can provide random shuffling on a character-by-character, word-by-word, or utterance-by-utterance basis. For stock prices and sales data, the present invention is useful when multiple different time resolutions, such as minutes, hours, days, months, and years, are to be handled in parallel. Furthermore, since the present invention includes intentional information loss processing by random shuffling, it can also be applied to tasks that require consideration of privacy protection and information confidentiality.

[0070] In the above, in order to express the contents of the present invention, the present specification has described the embodiments of the present invention with reference to the drawings. However, the present invention is not limited to the above-mentioned embodiments, and includes modifications and improvements that are obvious to a person skilled in the art based on the matters described in the present specification. [Explanation of symbols]

[0071] 10... Information processing device 11... Input unit 12...Preprocessing section 13...Feature extraction section 14...Division section 15...Shuffle section 16... Output section 16a... First output section 16b...Second output section 16c...Integration section 17…Statistical Machine Learning Models 18…Pre-Trained Models 20...Neural network structure 21...First learning layer 22...Shuffle layer 22a...Dividing section 22b...Shuffle section 23...Second learning layer 30...Neural network structure 31...First learning layer 32...First shuffle layer 33...Second learning layer 34…Third learning layer 35…Second shuffle layer 36…Integration layer

Claims

1. A first learning layer that extracts features from time series data; a shuffle layer including a division unit that divides a feature sequence of the time-series data extracted in the first learning layer into a plurality of blocks, and a shuffle unit that at least partially randomly rearranges the time-series order of the plurality of blocks; A second learning layer is provided that performs statistical machine learning on the feature sequence data obtained by the sorting. Information processing device.

2. The dividing unit further divides the feature amount sequence of the time series data into a plurality of higher-level blocks each including a plurality of the blocks; The shuffle unit rearranges the chronological order of the blocks in the higher-order block, or rearranges the chronological order of the higher-order block. The information processing device according to claim 1 .

3. The shuffle unit at least partially rearranges the chronological order of the higher-level block while maintaining the chronological order of the blocks in the higher-level block. The information processing device according to claim 2 .

4. The shuffle unit at least partially rearranges the chronological order of the blocks in the higher-level block while maintaining the chronological order of the higher-level block. The information processing device according to claim 2 .

5. The shuffle unit determines whether or not a certain block is to be rearranged and which other block is to be replaced with the certain block, based on a predetermined probability distribution. The information processing device according to claim 1 .

6. the dividing unit divides the plurality of blocks, each including a feature quantity sequence of the time series data, into a hierarchy; The shuffle unit determines whether or not to rearrange the time series order of the blocks, and a rearrangement method is determined for each layer. The information processing device according to claim 1 .

7. The shuffle layer outputs the original time series data in which the time series order of the features has not been rearranged together with the feature series data obtained by the rearrangement to the second learning layer, or outputs the merged feature series data and the original time series data to the second learning layer. The information processing device according to claim 1 .

8. A program for causing a computer to function as the information processing device according to claim 1.

9. A first learning step in which a first learning layer extracts features of time series data; a step of dividing a feature sequence of the time-series data extracted during the first learning step into a plurality of blocks by a division unit included in a shuffle layer; a shuffle unit included in the shuffle layer at least partially randomly rearranges the time series order of the plurality of blocks; The second learning layer includes a second learning step of performing statistical machine learning on the feature sequence data obtained by the sorting. A method for processing information using a computer.

10. A first learning step in which a first learning layer extracts features of time series data; a step of dividing a feature sequence of the time-series data extracted during the first learning step into a plurality of blocks by a division unit included in a shuffle layer; a shuffle unit included in the shuffle layer at least partially randomly rearranges the time series order of the plurality of blocks; The second learning layer includes a second learning step of performing statistical machine learning on the feature sequence data obtained by the sorting. How a computer produces trained models.

Images