RNN training apparatus, method, and program

Abstract

To provide a recurrent neural network (RNN) training device, method, and program that improve the stability and efficiency of the convergence of training of an RNN.SOLUTION: A recurrent neural network (RNN) training device 100 is provided with: a storage 2 that stores a hidden state that is intermediate output data of an RNN for every N series; and a processing circuit 1 having a construction unit 12 that selects data of M (smaller than N) series from the data of the N series used for training of the RNN to construct a mini batch and outputs series information specifying the selected series, a reading unit 1 that reads, from the storage, the hidden state before processing of a series corresponding to the series information according to the series information, an optimization unit 14 that executes an optimization operation on the RNN based on the hidden state before processing and mini batch, and a writing unit 15 that writes, in the storage based on the series information, the hidden state after processing that is intermediate output data of the RNN obtained through the optimization operation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Embodiments of the present invention relate to an RNN training apparatus, method, and program. [Background technology]

[0002] There is a method for constructing minibatches from time-series data of various sequence lengths and training a recurrent neural network (RNN) in units of these minibatches. In RNNs, there is a technique called TBPTT (Truncated Back Propagation Through Time). In TBPTT, instead of backpropagating errors through the sequence data indefinitely in the time direction during RNN training, errors are backpropagated in units of blocks cut out at fixed-length time steps (e.g., 128).

[0003] In RNNs, minibatches must be constructed and trained while ensuring the continuity of the time series by inheriting hidden states, etc. Furthermore, in training deep neural networks, it is necessary to shuffle the time series data supplied to the RNN in order to avoid data bias. When training an RNN with time series data in minibatches, shuffling in blocks delimited by TBPTT breaks the continuity of hidden states that should propagate beyond TBPTT. Therefore, when training an RNN with time series data in minibatches using TBPTT, if one tries to ensure the continuity of hidden states, bias will occur in the time series data supplied to the RNN, reducing the degree of freedom in selecting the time series data that make up the minibatches, leading to unstable convergence of training and decreased efficiency. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Viacheslav Khomenko, et al. “Accelerating Recurrent Neural Network Training using Sequence Bucketing and Multi-GPU Data Parallelization”

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the present invention is to provide an RNN training device, method, and program that enable improvement in the stability and efficiency of the convergence of recurrent neural network training.

Means for Solving the Problems

[0006] The RNN training device according to the embodiment includes a storage unit, a construction unit, a reading unit, an optimization unit, and a writing unit. The storage unit stores hidden states, which are intermediate output data of the recurrent neural network, for each of the N sequences. The construction unit selects M sequences of data fewer than N from the N sequences of data used for training the recurrent neural network to construct a mini-batch, and outputs sequence information identifying the selected sequences. The reading unit reads from the storage unit the hidden state before processing of the sequence corresponding to the sequence information according to the sequence information. The optimization unit executes an optimization operation of the recurrent neural network based on the hidden state before processing and the mini-batch. The writing unit writes the hidden state after processing, which is the intermediate output data of the recurrent neural network obtained by the optimization operation, into the storage unit according to the sequence information.

Brief Description of the Drawings

[0007] [Figure 1] Figure showing a configuration example of the RNN training device according to the present embodiment [Figure 2] Figure showing a structural example of sequence data [Figure 3] Figure showing an example of a processing procedure of the RNN training process [Figure 4] Figure 3 shows an example of the functional configuration of the RNN training process. [Figure 5] A diagram showing an example of the mini-batch construction process. [Figure 6] A schematic diagram illustrating the forward propagation calculation in optimization operations. [Figure 7] Figure 3 shows the processing procedure for Example 1, from the mini-batch initialization process (step SA2) to the mini-batch presence / absence determination process (step SA4). [Figure 8] Figure 3 shows the processing procedure for Example 2, from the mini-batch initialization process (step SA2) to the mini-batch presence / absence determination process (step SA4). [Figure 9] Figure 3 shows the processing procedure for Example 3, from the mini-batch initialization process (step SA2) to the mini-batch presence / absence determination process (step SA4). [Figure 10] This diagram schematically shows the RNN used in the effectiveness verification according to this embodiment. [Figure 11] A graph showing test scores when the batch size is 16. [Figure 12] A graph showing test scores when the batch size is 24. [Figure 13] A schematic diagram showing an RNN related to a modified example. [Figure 14] Diagram illustrating the method used in the comparative example. [Modes for carrying out the invention]

[0008] The RNN training apparatus, method, and program according to this embodiment will be described below with reference to the drawings.

[0009] Figure 1 shows an example configuration of the RNN training device 100 according to this embodiment. As shown in Figure 1, the RNN training device 100 is a computer having a processing circuit 1, a storage device 2, an input device 3, a communication device 4, and a display device 5. Data communication between the processing circuit 1, the storage device 2, the input device 3, the communication device 4, and the display device 5 is performed via a bus. The RNN training device 100 trains a recurrent neural network (RNN).

[0010] The processing circuit 1 includes a processor such as a CPU (Central Processing Unit) and memory such as RAM (Random Access Memory). The processing circuit 1 includes an acquisition unit 11, a construction unit 12, a reading unit 13, an optimization unit 14, a writing unit 15, and a learning control unit 16. The processing circuit 1 implements each of the above units 11 to 16 by executing an RNN training program. The RNN training program is stored in a non-temporary computer-readable storage medium such as a storage device 2. The RNN training program may be implemented as a single program describing all the functions of the above units 11 to 16, or it may be implemented as multiple modules divided into several functional units. Furthermore, each of the above units 11 to 16 may be implemented by an integrated circuit such as an Application Specific Integrated Circuit (ASIC). In this case, it may be implemented in a single integrated circuit, or it may be implemented individually in multiple integrated circuits.

[0011] The acquisition unit 11 acquires N sequences of data (hereinafter referred to as sequence data) to be used for training the RNN. Sequence data includes multiple elements that conform to an arbitrary rule. Examples of sequence data according to this embodiment include time-series data containing multiple elements that conform to a time series, and language data containing multiple elements that conform to a word order. Time-series data is, for example, data whose elements are multiple measured values ​​output continuously from various measuring instruments. Language data is data whose elements are multiple words arranged in word order.

[0012] The construction unit 12 selects M (a natural number) sequences from the N sequence data used to train the RNN, constructs a mini-batch, and outputs sequence information that identifies the selected sequences. The number of sequences M that make up each mini-batch is called the mini-batch size. The sequence information uses an identifier that uniquely identifies the sequence.

[0013] The reading unit 13 reads the pre-processing hidden state of the sequence corresponding to the sequence information from the storage device 2 according to the sequence information. The reading of the pre-processing hidden state is performed before the optimization calculation by the optimization unit 14. The hidden state represents the intermediate output data from the RNN based on the sequence data. The pre-processing hidden state represents the hidden state that will be used for the optimization calculation by the optimization unit 14.

[0014] The optimization unit 14 performs RNN optimization calculations based on the pre-processing hidden state and the mini-batch. In the optimization calculations, the optimization unit 14 performs forward propagation calculations, backpropagation calculations, and parameter updates. In the forward propagation calculations and / or backpropagation calculations, the optimization unit 14 calculates the post-processing hidden state. The post-processing hidden state refers to the hidden state obtained through the optimization calculations.

[0015] The writing unit 15 writes the processed hidden state, which is the intermediate output data of the RNN obtained by the optimization calculation, to the storage device 2 according to the sequence information.

[0016] The learning control unit 16 controls the training process of the RNN. The learning control unit 16 determines whether the update termination condition is met and controls the acquisition unit 11, construction unit 12, reading unit 13, optimization unit 14, and writing unit 15 to repeat the training process until it is determined that the condition is met. If the learning control unit 16 determines that the update termination condition is met, it terminates the training process.

[0017] Storage device 2 consists of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), integrated circuit storage devices, etc. Storage device 2 stores RNN training programs, etc. In addition, storage device 2 stores hidden states for each N sequence in a read-write manner.

[0018] Input device 3 receives various commands from the user. Input device 3 can include a keyboard, mouse, various switches, touchpad, touch panel display, etc. Output signals from input device 3 are supplied to processing circuit 1. Input device 3 may also be an input device of a computer connected to processing circuit 1 via wired or wireless connection.

[0019] Communication device 4 is an interface for data communication with external devices connected to the RNN training device 100 via a network.

[0020] Display device 5 displays various information. As the display device 5, a CRT (Cathode-Ray Tube) display, liquid crystal display, organic EL (Electro-Luminescence) display, LED (Light-Emitting Diode) display, plasma display, or any other display known in the art can be used as appropriate. Alternatively, the display device 5 may be a projector.

[0021] The following describes an example of the operation of the RNN training device 100 according to this embodiment.

[0022] First, the structure of the sequence data according to this embodiment will be explained with reference to Figure 2. Figure 2 is a diagram showing an example of the structure of sequence data. As shown in Figure 2, the sequence data contains multiple elements that conform to an arbitrary rule. The multiple elements are divided into K (natural number) blocks. Each block contains n (natural number) elements that conform to an arbitrary rule. The number of elements n in each block is a fixed value called the TBPTT length, regardless of the block. That is, the sequence length is the same across the K blocks. The sequence data will contain K × n elements. The division of the sequence data into K blocks may be performed by the construction unit 12, or the sequence data divided into K blocks may be acquired by the acquisition unit 11. The number of blocks in each sequence data may be the same for each sequence or may be different.

[0023] Next, the RNN training process using the RNN training device 100 according to this embodiment will be described.

[0024] Figure 3 shows an example of the processing procedure for RNN training. Figure 4 shows an example of the functional configuration of the RNN training process shown in Figure 3. It is assumed that N time-series data have already been acquired by the acquisition unit 11 at the start of Figure 3.

[0025] First, the learning control unit 16 sets index i to the value "0" (step SA1). Index i is a variable that represents the number of epochs used to determine the update completion condition.

[0026] When step SA1 is performed, the learning control unit 16 initializes the hidden state storage area 21 (step SA2). The hidden state storage area 21 is a storage area for hidden states provided in the storage device 2.

[0027] When step SA3 is performed, the construction unit 12 selects M series data from N series data 41 to construct a mini-batch 42 and outputs series information 43 that identifies the selected series (step SA3).

[0028] Figure 5 shows an example of the mini-batch construction process. In Figure 5, there are 6 sequences, q, r, s, t, u, and v, i.e., N=6, and the mini-batch size M is 3. Each sequence is divided into multiple blocks according to the sequence order, and each block has a TBPTT length. For example, if the sequence data is time-series data, the sequence order is time. The number of blocks included in each sequence does not have to be the same; as shown in Figure 5, they may be different. For example, sequence q contains 5 blocks, while sequence r contains 3 blocks. Note that one mini-batch 42m (where m is the mini-batch index, and 1 ≤ m ≤ 8) only needs to be constructed for each step of mini-batch training. Hereafter, one step of mini-batch training will be referred to as a time step.

[0029] The construction unit 12 selects three blocks from six sequence data. The three selected blocks constitute one mini-batch 42m. There are various ways to select the three blocks. For example, first, the construction unit 12 randomly selects three sequence data from six sequence data. Alternatively, the construction unit 12 may select three sequence data from six sequence data according to a predetermined rule. Next, for each of the selected sequence data, the construction unit 12 sequentially selects the unprocessed blocks. Specifically, the construction unit 12 selects the unprocessed blocks in order from the earliest to the latest sequence order. For example, for the first mini-batch 421, block s0 from sequence data s, block q0 from sequence data q, and block u0 from sequence data u are selected.

[0030] Furthermore, when a block is selected, the construction unit 12 outputs the series information of the selected block. For example, if block s0 from series data s, block q0 from series data q, and block u0 from series data u are selected as the first mini-batch 421, the unit outputs identifiers representing series s, series q, and series u. For example, the construction unit 12 maintains a series information database, and when a block is selected, it queries the series information database with the series data of that block and outputs the series information corresponding to that block. The series information database is a database that systematically associates series information for each type of series data. The series information only needs to include at least the series identifier.

[0031] When step SA3 is performed, the learning control unit 16 determines whether there are any unprocessed minibatches (step SA4). The presence or absence of unprocessed minibatches can be determined based on the presence or absence of unselected blocks. Specifically, the learning control unit 16 determines that there are unprocessed minibatches if there are unselected blocks, and determines that there are no unprocessed minibatches if there are no unselected blocks.

[0032] If it is determined in step SA4 that there are unprocessed minibatches (step SA4: YES), the read unit 13 reads the pre-processing hidden state 44 from the hidden state storage area 21 according to the sequence information 43 output in step SA3 (step SA5). The hidden state storage area 21 is a storage area provided in the storage device 2. The hidden state storage area 21 is allocated for each sequence of data and stores hidden information related to that sequence of data. The read unit 13 searches the hidden state storage area 21 using the sequence information 43 as an index and reads the hidden state (pre-processing hidden state) from the storage area of ​​the sequence information 43. Note that the hidden state is overwritten in the hidden state storage area 21, so for each sequence of data, only the latest hidden state, i.e., the hidden state obtained by the previous optimization operation, is stored.

[0033] In step SA5, the optimization unit 14 executes an optimization operation based on the mini-batch 42 constructed in step SA3 and the pre-process hidden state 44 read in step SA5 (step SA6). In the optimization operation, the optimization unit 14 performs forward propagation calculation, backpropagation calculation, and parameter update. In the forward propagation calculation, the inputs and outputs of each layer are calculated. In the backpropagation calculation, the gradients of the calculated inputs and outputs of each layer are calculated. In the parameter update, the parameters are updated based on the calculated gradients. In the forward propagation calculation, the optimization unit 14 calculates the hidden state of the last forward propagation calculation output (the forward propagation calculation output from the n-th RNN) of the RNN layer. The hidden state is stored in the hidden state storage area 21. Hereinafter, the hidden state is referred to as the post-process hidden state. Note that the hidden state obtained by the forward propagation calculation performed after the parameter update may be stored in the hidden state storage area 21 as the "post-process hidden state".

[0034] FIG. 6 is a diagram schematically showing the forward propagation calculation in the optimization operation. In FIG. 6, it is assumed that the mini-batch size M is 3, and the mini-batch is constructed by the third block u2 of the series data u, the second block v1 of the series v, and the fifth block v4 of the series q. It is assumed that each block contains n elements x. That is, the TBPTT length is n. The third block u2 of the series data u contains n elements from element x u,2n to element x u,3nー1 ; the second block v1 of the series data v contains n elements from element x v,1n to element x v,2nー1 ; and the fifth block q4 of the series data q contains n elements from element x q,4n to element x q,5nー1 . The pre-process hidden state is represented by h u,2n-1 , h v,1n-1 , h q,4n-1 . The post-process hidden state is represented by h u,3n-1 , h v,2n-1 , h q,5n-1 .

[0035] The optimization unit 14 uses the pre-process hidden states h u,2n-1 , h v,n-1h q,4n-1 and n elements x u,2n ~x u,3nー1 , element x v,1n ~x v,2nー1 , element x q,4n ~x q,5nー1 The forward propagation calculation is performed by recursively applying the RNN60 to the RNN60. More specifically, as shown in Figure 6, the optimization unit 14 first processes the hidden state h before processing. u,2n-1 h v,1n-1 h q,4n-1 and the first element x u,2n , x v,2n , x q,4n Applying this to the first RNN, the first output y u,2n , y v,1n , y q,4n and the first intermediate hidden state h u,2n h v,1n h q,4n The output is then calculated by the optimization unit 14. u,2n h v,1n h q,4n and the next second element x u,2n+1 , x v,1n+1 , x q,4n+1 Applying this to the second RNN, we get the second output y u,2n+1 , y v,n+1 , y q,4n+1 and the second intermediate hidden state h u,2n+1 h v,1n+1 h q,4n+1 It outputs the following. Similarly, it applies the elements of the current time step and the hidden state obtained in the previous time step to the RNN of the current time step in order, and outputs the output of the current time step and the hidden state. Then the hidden state h of the (n-1)th time step is output. u,3n-2 h v,2n-2 h q,5n-2 and the nth element x u,3n-1 , x v,2n-1 , x q,5n-1 Applying this to the nth RNN, we get the nth output y u,3n-1 , y v,2n-1 , y q,5n-1 and the hidden state h after processing u,3n-1 h v,2n-1 h q,5n-1 It outputs the following.

[0036] When step SA6 is performed, the writing unit 15 writes the processed hidden state 45 obtained by the optimization calculation in step SA6 to the hidden state storage area 21 according to the sequence information 43 output in step SA3 (step SA7). The writing unit 15 searches the hidden state storage area 21 using the sequence information 43 of each of the three sequence data included in the mini-batch 42 to be processed as an index, and overwrites the storage area of ​​the sequence information 43 with the processed hidden state 45. As a result, the latest hidden state for each of the sequence data is stored in the hidden state storage area 21.

[0037] When step SA7 is performed, in step SA3 the construction unit 12 selects M sequence data from the unprocessed sequence data 41 to construct the next minibatch 42 and outputs the next sequence information 43. A read operation (step SA5), an optimization operation (step SA6), and a write operation (step SA7) are performed on the next minibatch 42 and / or the next sequence information 43.

[0038] In this manner, steps SA3 to SA7 are repeated as described above until it is determined in step SA4 that there are no unprocessed mini-batches 42. Note that, as shown in Figure 5, the last mini-batch 428 may have a smaller mini-batch size, i.e., fewer blocks than the other mini-batches 421 to 427. In this case, mini-batch 428 may be padded with predetermined blocks to match the number of blocks in mini-batch 428 to that of the other mini-batches 421 to 427. Also, in step SA3, when it is determined that there are no unprocessed mini-batches 42, no mini-batch 42 is constructed, and no sequence information 43 is output.

[0039] If it is determined in step SA4 that there are no unprocessed mini-batches 42 (step SA4: NO), the learning control unit 16 adds the value "1" to index i and determines whether index i is below the upper limit of epochs TH (step SA8). The upper limit of epochs TH can be set to any value according to experience or any algorithm.

[0040] If, in step SA8, it is determined that index i is below the upper limit of epoch number TH (step SA8: YES), then for the next epoch, steps SA1 to SA7 are repeated until, in step SA4, it is determined that there are no unprocessed mini-batch 42s.

[0041] Then, in step SA8, if it is determined that index i does not fall below the upper limit of epochs TH (step SA8: NO), the learning control unit 16 outputs the trained network parameters 46 (step SA9). The trained network parameters 46 are stored in the memory device 2. The trained network parameters 46 are assigned to the RNN to constitute the trained RNN.

[0042] With the above steps completed, the RNN training process by the RNN training device 100 is finished.

[0043] The RNN training process shown in Figures 3 and 4 is just one example, and various modifications are possible.

[0044] Next, a detailed example of the mini-batch initialization process (step SA2) to the mini-batch presence / absence determination process (step SA4) shown in Figure 3 will be described.

[0045] <Example 1> Figure 7 shows the processing procedure of Example 1, from the mini-batch initialization process (step SA2) shown in Figure 3 to the process of determining whether or not a mini-batch exists (step SA4). Step SB1 is provided between steps SA2 and SA3, steps SB2 to SB3 correspond to step SA3, and steps SB4 to SB6 correspond to step SA4. The construction unit 12 in Example 1 randomly selects M sequences from among N sequences, each having a remaining length of 1 or more.

[0046] As shown in Figure 7, the learning control unit 16 creates a dictionary seq_len, which has the sequence identifier id as the key and the sequence length as the value, and a dictionary remain_len, which has the sequence identifier id as the key and the remaining length as the value (step SB1). The dictionaries seq_len and remain_len are created for all sequences.

[0047] When step SB1 is performed, the construction unit 12 randomly selects a mini-batch size (M sequences) from the dictionary remain_len (step SB2). Step SB2 randomly selects M sequences from among the N sequences whose remaining length is 1 or greater.

[0048] When step SB2 is performed, the construction unit 12 extracts a block with an offset of seq_len[id]-remain_len[id] from the sequence selected in step SB2 (selected sequence), constructs a minibatch, and outputs sequence information (step SB3). seq_len[id] is a dictionary of sequence identifiers id that outputs the sequence length of the selected sequence. remain_len[id] is a dictionary of sequence identifiers id that outputs the remaining length of the selected sequence. seq_len[id]-remain_len[id] represents the position of the block selected in the current time step within the sequence data.

[0049] When step SB3 is performed, the learning control unit 16 subtracts the TBPTT length from the remaining length of the selected sequence (step SB4). That is, in step SB4, the learning control unit 16 executes remain_len[id] - = TBPTT_length. TBPTT_length represents the TBPTT length.

[0050] When step SB4 is performed, the learning control unit 16 removes the selection identifier id from the dictionary remain_len if remain_len[id] is 0 or less (step SB5). remain_len[id] being 0 or less means that there are no remaining blocks in that sequence. In this case, there is no need to select it in the remaining time steps, so the selection identifier id for that sequence is removed from the dictionary remain_len.

[0051] When step SB5 is performed, the learning control unit 16 determines whether or not there is content in the dictionary remain_len (step SB6). If there is content in the dictionary remain_len, it means that there is a sequence with remaining blocks, and if there is no content in the dictionary remain_len, it means that there is no sequence with remaining blocks.

[0052] If it is determined in step SB6 that the dictionary remain_len contains data (step SB6: NO), the read operation (step SA5), optimization operation (step SA6), and write operation (step SA7) shown in Figure 3 are executed, and the process returns to step SB2. If it is determined in step SB6 that the dictionary remain_len contains no data (step SB6: YES), the determination process (step SA8) shown in Figure 3 is performed. Steps SA1 to SA7 are repeated until it is determined that the number of epochs (index i) exceeds the upper limit of the number of epochs TH.

[0053] This concludes Example 1.

[0054] <Example 2> Figure 8 shows the processing procedure of Embodiment 1, from the mini-batch initialization process (step SA2) shown in Figure 3 to the mini-batch presence / absence determination process (step SA4). Step SC1 is provided between steps SA2 and SA3, steps SC2 to SC3 correspond to step SA3, and steps SC4 to SC6 correspond to step SA4. Steps SC1, 3 to 6 are the same as steps SB1, 3 to 6, so their explanation is omitted. The construction unit 12 in Embodiment 2 randomly selects a number of first sequences from the M sequences, corresponding to the product of the mini-batch size M and the selectivity, and for the remaining second sequences, prioritizes selecting sequences with longer remaining lengths.

[0055] As shown in Figure 8, when step SC1 is performed, the construction unit 12 randomly selects a series of mini-batch size (M series) * α, and then selects a series of mini-batch size (M series) * (1-α) from the dictionary remain_len in descending order of remaining length (step SB2). The parameter α is 0 < α < 1 and corresponds to the random selection rate. When there is a large variation in series lengths and only long series remain, reducing α can reduce padding.

[0056] This concludes Example 2.

[0057] <Example 3> Figure 9 shows the processing procedure of Example 3, from the mini-batch initialization process (step SA2) shown in Figure 3 to the process of determining whether or not a mini-batch exists (step SA4). Step SD1 is provided between steps SA2 and SA3, steps SD2 to SD3 correspond to step SA3, and steps SD4 to SD6 correspond to step SA4. Steps SCD, 3 to 6 are the same as steps SB1, 3 to 6, so their explanation is omitted. The construction unit 12 in Example 3 randomly selects M sequences from among N sequences, prioritizing sequences with a small difference between the sequence length and the remaining length.

[0058] As shown in Figure 9, when step SD1 is performed, the construction unit 12 randomly selects a mini-batch (M) of sequences, prioritizing sequences with a small (seq_len[id] - remain_len[id]) ratio (step SD2). (seq_len[id] - remain_len[id]) represents the sequence length of the processed block. This ensures that sequences are selected evenly across multiple sequences, making it possible to select multiple sequences so that the time step offsets are aligned.

[0059] This concludes Example 3.

[0060] (effect) The effects of this embodiment will be explained with reference to Figures 10, 11, and 12. Figure 10 is a schematic diagram showing the RNN used to verify the effects of this embodiment. As shown in Figure 10, the RNN is assumed to be a temperature estimation (regression) problem with input elements x having 91 dimensions, a TBPTT length of 128, a hidden state h having 8 dimensions, and an output y having 4 dimensions. The RNN calculates the hidden state h for the next time step based on elements x and the hidden state h, and calculates the output y by applying a linear transformation layer (Linear) to the hidden state h.

[0061] Figure 11 shows the test scores when the batch size is 16. In the left graph of Figure 11, the vertical axis is the logarithm of the test score, MSE (mean squared error), and the horizontal axis is the sample number. MSE is the value obtained by dividing the sum of the squares of the differences between the predicted value and the correct value by the number of data points. A lower MSE indicates better accuracy of the RNN. The sample number is the number assigned to the combination of learning rate, weight decay, Gaussian noise, and random seed. The right graph of Figure 11 is a graph sorted by the MSE value. In the left and right graphs of Figure 11, the solid lines represent the method according to this embodiment, and the dotted lines represent the method according to the comparative example. The method according to the comparative example is the method according to Non-Patent Document 1 (Viacheslav Khomenko, et al. “Accelerating Recurrent Neural Network Training using Sequence Bucketing and Multi-GPU Data Parallelization”), i.e., bucketing.

[0062] Figure 12 shows the test scores when the batch size is 24. In the left graph of Figure 12, the vertical axis is the logarithm of the test score (MSE), and the horizontal axis is the sample number. The sample number is the number assigned to the combination of learning rate, weight decay, Gaussian noise, and random seed. The right graph of Figure 12 is a graph sorted by the MSE value. In the left and right graphs of Figure 12, the solid lines represent the results of this embodiment, and the dotted lines represent the results of the comparative example.

[0063] Here, we will briefly explain the comparative example with reference to Figure 14. As shown in Figure 14, in the comparative example, as in this embodiment, six sequence data sets, q, r, s, t, u, and v, are prepared, and the sequence length of each sequence is assumed to be the same. The mini-batch size is also assumed to be M=3, as in this embodiment. In the comparative example, hidden state reading and writing are not performed. In the comparative example, the six sequence data sets are sorted in descending order by sequence length. Then, bucketing and padding are performed. That is, the six sequence data sets are divided into two buckets with similar sequence lengths. The empty blocks in each bucket are padded. Then, three blocks of three sequence data sets are selected from the buckets in ascending order of their numbers to construct a mini-batch. In the comparative example, since hidden state reading and writing are not performed, in order to inherit the hidden state of the RNN, the sequences of data blocks must be placed consecutively into the same slots in the mini-batch. For this reason, the degree of freedom in selecting the sequence data sets that make up the mini-batch is reduced, and the convergence of training may become unstable. Also, because padding is performed, the number of mini-batches increases, which may reduce training efficiency.

[0064] On the other hand, the RNN training device 100 according to this embodiment has a read unit 13, a write unit 15, and a hidden state storage area 21, so that the hidden state can be read and written at any time. Therefore, this embodiment can reduce padding compared to the comparative example, and thus reduce computational unusable blocks due to padded blocks. In addition, this embodiment can make the type of sequence data or blocks different between adjacent minibatches compared to the comparative example, so the bias in the training data is reduced, the convergence of training is stabilized, and the performance of the final converged RNN is improved. As shown in Figures 11 and 12, this embodiment generally has a lower MSE than the comparative example. Therefore, it can be seen that the accuracy of the RNN in this embodiment is better than that of the comparative example.

[0065] (modified version) Figure 13 schematically shows an RNN relating to a modified example. As shown in Figure 13, the RNN may have two layers. In this case, the first RNN takes a hidden state h and a 91-dimensional element x as input and outputs an intermediate output and an 8-dimensional hidden state h. The second RNN takes the intermediate output and a hidden state k as input and outputs an intermediate output and a 4-dimensional hidden state k. The intermediate output is scaled by a scale layer and converted into a 4-dimensional output y. Note that the number of dimensions for each data is just an example and is not limited to this. Also, the number of layers in the RNN may be three or more. In addition, other networks such as a linear transformation layer may be included instead of or in addition to the scale layer.

[0066] Thus, according to this embodiment, it is possible to provide an RNN training device, method, and program that enable improvements in the stability and efficiency of convergence during training of a recurrent neural network.

[0067] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0068] 1... Processing circuit, 2... Memory device, 3... Input device, 4... Communication device, 5... Display device, 11... Acquisition unit, 12... Construction unit, 13... Read unit, 14... Optimization unit, 15... Write unit, 16... Learning control unit, 100... RNN training device.

Claims

1. A memory unit that stores the hidden state, which is the intermediate output data of the recurrent neural network, for each of the N sequences, A construction unit that selects M data sequences (fewer than N) from the N data sequences used to train the recurrent neural network, constructs a mini-batch, and outputs sequence information that identifies the selected sequences. A reading unit reads the hidden state of the sequence corresponding to the sequence information before processing from the storage unit, according to the sequence information. An optimization unit that performs an optimization operation of the recurrent neural network based on the hidden state before processing and the minibatch, A writing unit writes the processed hidden state, which is the intermediate output data of the recurrent neural network obtained by the optimization calculation, to the storage unit according to the sequence information. An RNN training device equipped with the following.

2. The RNN training apparatus according to claim 1, wherein the construction unit randomly selects M sequences from the N sequences.

3. The RNN training apparatus according to claim 2, wherein the construction unit randomly selects M sequences from among the N sequences, each sequence having a remaining length of 1 or more.

4. The RNN training apparatus according to claim 3, wherein the construction unit randomly selects a number of first sequences from the M sequences corresponding to the product of the mini-batch size and the selection rate, and for the remaining second sequences, it prioritizes selecting sequences with longer remaining lengths.

5. The RNN training apparatus according to claim 1, wherein the construction unit randomly selects M sequences from among the N sequences, prioritizing sequences in which the difference between the sequence length and the remaining length is small.

6. The RNN training apparatus according to claim 1, wherein each of the N sequences of data is divided into blocks having a common TBPTT length.

7. The RNN training apparatus according to claim 6, wherein the construction unit, for each of the M sequences, selects the unprocessed blocks from the selected sequences in order when selecting the data for the M sequences.

8. The RNN training device according to claim 1, wherein the optimization unit performs forward propagation calculation, backpropagation calculation and parameter update in the optimization calculation, and calculates the hidden state in the forward propagation calculation and / or backpropagation calculation.

9. The RNN training apparatus according to claim 1, wherein the sequence information has an identifier for the selected sequence.

10. The RNN training device according to claim 1, wherein the writing unit overwrites the hidden state before processing with the hidden state after processing.

11. A mini-batch is constructed by selecting M data sequences (fewer than N) from the N data sequences used to train a recurrent neural network, and outputting sequence information that identifies the selected sequences. According to the aforementioned sequence information, read the hidden state of the sequence corresponding to the sequence information before processing from the storage unit. Based on the hidden state before processing and the mini-batch, the optimization operation of the recurrent neural network is performed. The processed hidden state, which is the intermediate output data of the recurrent neural network obtained by the optimization calculation, is written to the storage unit according to the sequence information. A RNN training method that includes the following.

12. On the computer, A function to select M data sequences (fewer than N) from N data sequences used to train a recurrent neural network, construct a mini-batch, and output sequence information that identifies the selected sequences. A function to read the hidden state of the sequence corresponding to the sequence information before processing from the storage unit, according to the sequence information, A function to perform an optimization operation on the recurrent neural network based on the hidden state before processing and the minibatch, A function to write the processed hidden state, which is the intermediate output data of the recurrent neural network obtained by the optimization calculation, to the storage unit according to the sequence information, An RNN training program that makes this possible.

Images