Recurrent neural network system, program, information processing method, information processing device, and learning method
The recurrent neural network system addresses error accumulation in RNNs by using CRNN for synchronization and PRNN for prediction, achieving precise long-term time series forecasting through residual control and optimized parameter adjustment.
Patent Information
- Application Number
- JP2023080371
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-27
AI Technical Summary
Recurrent Neural Networks (RNNs) experience error accumulation during long-term time series prediction due to the approximation of state variables, making it difficult to learn the time evolution of state variables and data output with high precision.
A recurrent neural network system that incorporates external force data, hidden layer state variables, and measured data to determine the hidden layer state variable at a second time point, using CRNN for synchronization and PRNN for prediction, with error minimization through residual control and optimized parameter adjustment.
The system effectively suppresses error accumulation, enabling accurate long-term time series prediction by maintaining synchronization with actual phenomena and minimizing long-term prediction errors.
Smart Images

Figure 2024164700000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a recurrent neural network system, a program, an information processing method, an information processing device, and a learning method for predicting data to be obtained at the next or subsequent time from given time series data. [Background technology]
[0002] Recurrent Neural Networks (RNNs) have been proposed as a learning model that learns the generation process of time series data and predicts future data when past time series data is input (for example, Patent Document 1). RNNs have a recursive structure that reflects past states in the current state. RNNs calculate the next state by simultaneously considering the input data and the previous state, so they can be used for time series prediction. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-282298 Summary of the Invention [Problem to be solved by the invention]
[0004] However, in an RNN, the time evolution of the state variables (hidden layer state variables) that define the state is an approximation of the time evolution of real phenomena, so when time series prediction is performed repeatedly, errors accumulate in the RNN state variables. For this reason, in the learning stage of an RNN aimed at long-term future prediction, it is difficult to learn with high accuracy the time evolution of the state variables and the data output process.
[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a recurrent neural network system, a program, an information processing method, an information processing device, and a learning method that suppress the accumulation of errors. [Means for solving the problem]
[0006] A recurrent neural network system according to one embodiment of the present application receives external force data at a second time point, hidden layer state variables at a first time point prior to the second time point, and an error between actual data at the first time point and predicted data at the first time point, and calculates hidden layer state variables at the second time point, and calculates predicted data for the second time point based on the hidden layer state variables at the second time point. Effect of the Invention
[0007] According to one aspect of the present application, even when performing long-term time series prediction, it is possible to model the generation process of time series data and perform future prediction while suppressing accumulation of errors. [Brief description of the drawings]
[0008] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a recurrent neural network system. [Diagram 3] 13 is a flowchart illustrating an example of a learning process procedure. [Figure 4] 13 is a flowchart illustrating an example of a procedure for a prediction process. [Diagram 5] FIG. 2 is an explanatory diagram showing an example of an observation quantity DB. [Figure 6] FIG. 11 is an explanatory diagram showing an example of a parameter DB. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, the embodiments will be described with reference to the drawings. The system to be covered in this specification will be described. The system to be covered in this specification is one whose state change conforms to the following formula (10).
[0010] h t =F(h t-1 , a t ) (t=1, 2, 3,…) … (10)
[0011] Here, ht is the total state of the system at time t, a t represents the external force (external force data) on the system. The function F defines the time evolution of this system. In this system, the measured data u t The function G is a function that determines the actual measurement data according to the state of the system.
[0012] u t =G(h t ) … (11)
[0013] The algorithm described in this specification uses the measured data u t and external force data a t Given time series data (t=1, 2, 3, ...), future predictions are made by constructing approximation functions for functions F and G (or functions that are homeomorphic to them). Details will be described later.
[0014] 1 is a block diagram showing an example of the hardware configuration of an information processing device. The information processing device 1 is composed of a server computer, a workstation, a PC (Personal Computer), etc. The information processing device 1 includes a control unit 11, a main memory unit 12, an auxiliary memory unit 13, an input unit 14, a display unit 15, a communication unit 16, and a reading unit 17. Each component is connected by a bus B. The information processing device 1 may be composed of a multicomputer consisting of multiple computers, a virtual machine virtually constructed by software, or a quantum computer. The functions of the information processing device 1 may also be realized by a cloud service.
[0015] The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), etc. The control unit 11 reads out and executes a control program 1P (program, program product) stored in the auxiliary storage unit 13, thereby performing various information processing, control processing, etc., and realizing functional units such as an acquisition unit and an output unit.
[0016] The main memory unit 12 is a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, etc. The main memory unit 12 mainly temporarily stores data necessary for the control unit 11 to execute arithmetic processing.
[0017] The auxiliary storage unit 13 is a hard disk or a solid state drive (SSD) or the like, and stores a control program 1P and various databases (DBs) necessary for the control unit 11 to execute processing. The auxiliary storage unit 13 also stores a synchronization recurrent neural network and a prediction recurrent neural network. The auxiliary storage unit 13 stores an observable DB 131 and a parameter DB 132. The auxiliary storage unit 13 may be separate from the information processing device 1 and an external storage device externally connected to the information processing device 1. The various DBs and the like stored in the auxiliary storage unit 13 may be stored in a database server or cloud storage different from the information processing device 1.
[0018] The input unit 14 is a keyboard, a mouse, etc. The display unit 15 is configured with a liquid crystal display panel, an organic EL (Electro Luminescence) display panel, etc. The input unit 14 and the display unit 15 may be integrated to configure a touch panel display. Note that the information processing device 1 may display on an external display device (display device).
[0019] The communication unit 16 communicates with other computers via a network such as the Internet or a public communication network. The control unit 11 may use the communication unit 16 to download the control program 1P from another computer via a network or the like and store it in the auxiliary storage unit 13.
[0020] The reading unit 17 reads the portable storage medium 1a including a CD (Compact Disc)-ROM and a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 17 and store it in the auxiliary storage unit 13. The control unit 11 may also read the control program 1P from the semiconductor memory 1b.
[0021] FIG. 2 is an explanatory diagram showing an example of the configuration of a recurrent neural network system. The recurrent neural network system has a CRNN (recurrent neural network for synchronization) and a PRNN (recurrent neural network for prediction). The CRNN and PRNN are, for example, configured with a recurrent neural network (RNN). FIG. 2 shows the period from time t to time t+p for each of the recurrent neural networks CRNN and PRNN, and shows how information about time series data propagates through the network. p represents the required prediction period.
[0022] The measured data at time t (first time point) is expressed as u t , the external force data is a t The state variables (hidden layer state variables) of the CRNN at time t are expressed as h t * and the state is expressed as an argument to the function G(h t * ) with a hat t * is called the ex-post prediction data. t * is a function F(h t-1 * ,a t , δ t-1 * ) where δ t * is the error of the posterior prediction data against the actual data, and u t * -u t At time t, u t If is missing, δt * is set to 0. Therefore, the time interval of the CRNN does not need to match the time interval at which the actual measurement data is given. δ t * By correcting the state variables with the input of , the CRNN can learn to keep the state variables synchronized with the actual phenomenon for a long time. If the number of error components is not sufficient to correct the state variables, errors from multiple past times are used. For example, when using errors from two times, u with a hat t-1 * -u t-1 Ingredients and hats t * -u t The components of δ t * Let us assume that.
[0023] The PRNN generates prediction data for successive prediction periods. Let the state variable of the PRNN at time t be h t and use that state as an argument to calculate the function G(h t ) with prediction data and hat t is required. t is the function F(h t-1 , with hat t ,0) is determined by the hatched a t A with a hat t =a t Or it represents the predicted value of the external force data. If future external force data is available in the prediction stage, a with a hat is used in both the learning stage and the prediction stage. t is a t If future external force data cannot be obtained at the prediction stage, a with a hat is used at the prediction stage. t is the predicted value of the external force data, and in the learning stage, t At the end of the prediction period, the state variables of the PRNN are overwritten by the state variables of the CRNN at the same time, and prediction data for the next prediction period is generated.
[0024] The recurrent neural network of the present invention calculates the functions F and G and the initial value h0 of the state variables from the given time series data.* =h0. A specific example of the model is shown in the following equations (1) to (8). Equations (1) to (4) show the time evolution of the CRNN, and equations (5) to (8) show the time evolution of the PRNN.
[0025]
number
[0026]
number
[0027] In the learning stage, continuous input data (measurement data, external force data, etc.) for T time periods are input to the recurrent neural network of the present invention, and the network is propagated forward. At this time, the state variables of the PRNN are overwritten with the state variables of the CRNN for each required prediction period. In the example of Figure 2, in equation (5), s t =1.
[0028] The error δ of the CRNN obtained by the forward propagation t * and the error of the PRNN δ t The adjustment parameters are found by minimizing the loss function using the above. In the above model example, when squared error is used as the loss function, the adjustment parameters W0, W1, W2, W3, b0, b1, and b2, as well as the initial value h0 of the state variable (the value of the state variable at the initial time) are obtained using the following error minimization principle, equation (9). When the learning data contains multiple continuous time-series data, the sum of all errors (weighted sum error) is used as the loss function.
[0029]
number
[0030] To solve equation (9), a stochastic gradient algorithm (for example, the Adam method) is used. Learning is completed when the adjustment parameters W0, W1, W2, W3, b0, b1, and b2, as well as the initial value h0 of the state variable, are obtained. α represents the ratio of the errors of CRNN and PRNN, and is a parameter that assists optimization. The final goal is to minimize it at α=0, but it is easier to minimize it at α=1. For this reason, the value of α is adjusted in the range from 0 to 1 according to the progress of the gradient method. Most simply, α is fixed at α=0.5 (which treats the errors of CRNN and PRNN equally).
[0031] FIG. 3 is a flowchart showing an example of the procedure of the learning process. The control unit 11 of the information processing device 1 acquires learning data (step S1). The learning data is actually measured data (u t ) and external force data (a t ). The control unit 11 determines whether or not to use a predicted external force (= predicted value of external force data) (step S2). When the control unit 11 determines not to use a predicted external force (NO in step S2), the process proceeds to step S4. When the control unit 11 determines to use a predicted external force (YES in step S2), that is, when the input to the PRNN is a predicted value of the external force data, the control unit 11 calculates this predicted value (= predicted external force) (step S3). The present invention may be used for this calculation. The control unit 11 sets initial values of adjustment parameters (step S4). The control unit 11 calculates the forward propagation of the CRNN and the PRNN (step S5). The control unit 11 calculates the respective errors (δ t and δ t *) to obtain the value of the loss function (step S6). The control unit 11 judges whether or not to end the learning (step S7). When the control unit 11 judges that the minimization of the loss function value will not progress any further, it judges to end the learning. When the control unit 11 judges not to end the learning (NO in step S7), that is, when it judges that the minimization of the loss function value is necessary, it obtains a gradient related to the adjustment parameter of the loss function, updates the adjustment parameter according to a gradient method such as the Adam method (step S8), and returns the process to step S5. The control unit 11 repeats the processes from step S5 to step S8 to obtain the adjustment parameter that minimizes the loss function. When the control unit 11 judges that the learning will end (YES in step S7), it stores the adjustment parameter and the initial state variable in the auxiliary storage unit 13 (step S9), and ends the process.
[0032] In the prediction stage, the adjustment parameters for the time evolution function F of the state variables and the output function G of the prediction data are fixed and used as obtained in the learning stage. When making a prediction from time t+1 (next time point, second time point) to time t+p, only the initial value h0 of the state variables is obtained by optimizing the gradient method using the input data up to time t. This h0=h0 * Calculate the forward propagation of the CRNN from the state variable h t * Next, we set the initial value of the state variable h t =h t * Then, the PRNN is forward-propagated from time t+1 to time t+p to obtain predicted data from time t+1 to time t+p.
[0033] The following procedure is used for prediction after time t+p+1. The state variable h t * Then, the CRNN is forward propagated from time t+1 to time t+p, and the state variable h t+p * Next, we calculate the initial value of the state variable h t+p =h t+p *Then, the PRNN is propagated forward from time t+p+1 to time t+2p to obtain predicted data from time t+p+1 to time t+2p. The same process is repeated for predictions from time t+2p+1 onwards. The initial value h0 of the state variable can also be calculated again each time new input data is obtained.
[0034] FIG. 4 is a flowchart showing an example of the procedure of the prediction process. The control unit 11 acquires past input data (actual measurement data and external force data) (step S21). The control unit 11 fixes the parameters related to the state transition function and the predicted data output function to those calculated in the learning stage, and calculates the optimal initial value h0 of the state variables under the input data (step S22). The control unit 11 forward propagates the CRNN from the initial value of the state variables, and calculates the state variables at the latest time (step S23). The control unit 11 determines whether or not the external force data to be input to the PRNN can be acquired at a future time (step S24). If the control unit 11 determines that the external force data can be acquired (YES in step S24), the control unit 11 advances the process to step S26. If the control unit 11 determines that the external force data cannot be acquired (NO in step S24), the control unit 11 acquires predicted external force data (step S25). The control unit 11 inputs external force data or predicted external force data, propagates the PRNN forward for the required time (p time), and calculates predicted data (step S26). The control unit 11 judges whether or not to end (step S27). If the control unit 11 judges not to end (NO in step S27), after p time has elapsed, additional input data for this period is acquired (step S28). The control unit 11 judges whether or not recalculation of h0 is necessary (step S29). If the number of acquired past times is small, the control unit 11 judges that recalculation of h0 is necessary. If the control unit 11 judges that recalculation of h0 is necessary (YES in step S29), the process returns to step S22, and the optimization of h0 is redone, including the added input data. If the control unit 11 judges that recalculation of h0 is not necessary (NO in step S29), the process returns to step S23. If the control unit 11 judges that recalculation of h0 is necessary (YES in step S27), the process ends.
[0035] Next, a specific example of time series prediction using CRNN and PRNN will be described. As an example of time series prediction, the water level change of a certain river is predicted every hour. The water level is predicted every hour from midnight to 11pm every day based on input data until 11pm. The water level of a river is affected by the rainfall in the river basin and the tide level near the river mouth, so the external force data is the rainfall at multiple points in the river basin and the tide level near the river mouth. Since the observed amount of external force data cannot be used in the PRNN in the prediction stage, it is necessary to obtain predicted values for the rainfall and tide level. For the rainfall and tide level, it is possible to obtain predicted external forces by using the network of the present invention without input external force data. For example, the network of the present invention in which state variables are defined for each grid point is used under the assumption that the time evolution of the state variables and the output function of the predicted rainfall are only affected by nearby points, by using a convolution layer. In the learning stage, the rainfall and tide level input to the PRNN can be either actual data or predicted data. When modeling real phenomena is important, actual measured data is used, and when predictive performance is important, predicted data is used. The rainfall and tide levels input to the CRNN are the observed rainfall and tide levels in both the learning and prediction stages.
[0036] FIG. 5 is an explanatory diagram showing an example of an observation quantity DB. The observation quantity DB 131 stores time series data. The observation quantity DB 131 stores observation quantities corresponding to the observation quantities to be predicted and external forces. The observation quantity DB 131 includes a date column, a time column, a water level column, a rainfall column, and a tide level column. The date column stores the measurement date. The time column stores the measurement time. The water level column includes multiple columns. The name of each column is the name of a river water level observation station, and stores the water level at each observation station. The water level of a river means the height of the water surface of the river measured from the reference level. The elevation of the "reference level" is measured and determined in advance for each water level observation station. The rainfall column includes multiple columns. The name of each column is the name of the location where a rain gauge is installed, and stores the observed value of the rainfall at each location. Here, the rainfall for the previous hour is stored, but more instantaneous external force data may be used by storing the rainfall for the previous 10 minutes. The tide level column includes multiple columns. The name of each column is the name of an observation point, and the hourly tide level at each point is stored. For example, the hourly tide level is the smoothed value for each hour. The smoothed value is the tide level that removes components with periods up to about 3 hours, such as secondary vibrations, tsunamis, and waves, from the actual measured tide level.
[0037] In the observation amount DB131, the observation amount is used as learning data. The rainfall amount is assumed to be actual measurement data, and there is no external force data, and a rainfall prediction model is created using the RNN of the present invention. Similarly, for the tide level, a tide level prediction model is created from the tide level data. If there is missing data in the rainfall or tide level, the missing values are interpolated using the ex-post prediction data output from the CRNN. The interpolated rainfall and tide level are used as external force data, and the water level is actual measurement data, and a water level prediction model is created using the RNN of the present invention. The adjustment parameters of the state transition function and the predicted data output function in each model obtained by learning are saved. Thereafter, when making predictions at consecutive times, the state variables at the final time of the CRNN (for example, the state variables of the CRNN at 23:00 on March 31, 2023) are saved as the initial values of the state variables in the prediction stage.
[0038] FIG. 6 is an explanatory diagram showing an example of the parameter DB. Each cell of the parameter DB 132 generally includes multiple components. In the prediction stage, each parameter stored in the parameter DB 132 is used. For example, the predicted rainfall and predicted tide level for 24 hours from 0:00 to 23:00 on April 1, 2023 are obtained by forward propagating the PRNN of the rainfall model and the tide level model with h0 as the initial value for each. The predicted rainfall and predicted tide level obtained here are input to the PRNN of the water level model, and the predicted water level for 24 hours on the same day can be obtained by forward propagation. Next, after 23:00 on April 1, 2023, the actual measured values of the water level, rainfall, and tide level for 24 hours on the same day are obtained. The error between the predicted value and the actual measured value is input to the rainfall model and the tide level model, and the CRNN of each model is forward propagated to obtain the state variables as of 23:00 on April 1, 2023. The error between the predicted and actual values, as well as the actual rainfall and tide levels, are input to the water level model, and the state variables at the same time are obtained by forward propagation through the CRNN. Using these latest state variables as the initial values, it is possible to obtain predicted values for water levels, rainfall, and tide levels for 24 hours on April 2, 2023, in the same way as with the procedure performed on April 1, 2023. By repeating the same procedure thereafter, water level predictions can be made continuously for the next day.
[0039] This embodiment has the following advantages. The time evolution of the state variables of the CRNN is controlled using the error at the immediately preceding point in time (hereinafter, referred to as residual control). Specifically, in equation (1), W3δ t-1 *In a normal recurrent neural network, the measured data from the immediately preceding time point is often input. In comparison, residual control can keep the actual phenomenon and the model synchronized with fewer parameters and suppress the accumulation of errors in the state variables. In addition, when the residual is zero, it does not affect the time evolution of the state variables, and residual control of the state variables is performed only at the time when the measured data exists and the error is calculated. Therefore, if there is a missing value, no contradiction occurs by inputting zero. For this reason, it is easy to make the time evolution of the state variables shorter than the time interval of the measured data. In general, a shorter time interval makes learning easier because the state transition function is simpler. In addition, residual control cannot be performed by a CRNN alone during prediction, and short-term prediction is emphasized from the perspective of residual minimization, which often leads to excessive residual control. Therefore, by combining it with a PRNN that does not perform residual control and minimizing the long-term prediction error of the PRNN, it is possible to automatically adjust the degree of optimal residual control.
[0040] In addition, the error of the initial value h0 of the state variable is minimized by equation (9). Therefore, even if the training data is a set of short time-series data or the past input data given in the prediction stage is short, it is possible to build a model with high accuracy and make predictions.
[0041] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. The claims are written in a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this format. The claims may be written in a format in which multiple claims cite at least one other claim (multi-multi claim format). [Explanation of symbols]
[0042] 1: Information processing equipment 11: Control section 12: Main memory 13: Auxiliary storage section 131: Observation data DB 132: Parameter DB 14: Input section 15:Display section 16: Communications Department 17: Reading section 1P: Control program 1a: Portable storage medium 1b: Semiconductor memory B: Bus
Claims
1. 1. A recurrent neural network system, comprising: The external force data at the second time point, the hidden layer state variables at the first time point prior to the second time point, and the error between the actual measurement data at the first time point and the predicted data at the first time point are input, and the hidden layer state variables at the second time point are calculated, and the predicted data at the second time point is calculated based on the hidden layer state variables at the second time point. Recurrent neural network systems.
2. The hidden layer state variables at the initial time point are learned as part of the tuning parameters.
2. The recurrent neural network system according to claim 1.
3. At the next point in time when there is no actual measurement data, the input of the error is blocked, and the error becomes equal to zero.
3. The recurrent neural network system according to claim 1 or 2.
4. The system includes a recurrent neural network for synchronization that is trained using the error, and a recurrent neural network for prediction that is trained without using the error, At a predetermined time interval, the state variables of the prediction recurrent neural network are rewritten based on the state variables of the synchronization recurrent neural network.
3. The recurrent neural network system according to claim 1 or 2.
5. If no actual measurement data exists, the input to the synchronization recurrent neural network is blocked.
5. The recurrent neural network system according to claim 4.
6. The parameters are adjusted according to the error minimization principle (Equation 9) for the weighted sum error regarding the error from the synchronization recurrent neural network (Equation 1 to Equation 4) and the error from the prediction recurrent neural network (Equation 5 to Equation 8).
5. The recurrent neural network system according to claim 4. [0010] [0025] [0030]
7. Obtain time series data of actual measurement data, The external force data at the second time point, the hidden layer state variables at the first time point prior to the second time point, and the error between the actual measurement data at the first time point and the predicted data at the first time point are input, and the hidden layer state variables at the second time point are obtained, and the predicted data at the second time point is obtained based on the hidden layer state variables at the second time point. The time series data of the obtained actual measurement data is input to a recurrent neural network that has been trained to obtain the predicted data at the second time point based on the hidden layer state variables at the second time point, and the predicted data is output. A program that causes a computer to carry out processing.
8. The computer Obtain time series data of actual measurement data, The external force data at the second time point, the hidden layer state variables at the first time point prior to the second time point, and the error between the actual measurement data at the first time point and the predicted data at the first time point are input, and the hidden layer state variables at the second time point are obtained, and the predicted data at the second time point is obtained based on the hidden layer state variables at the second time point. The time series data of the obtained actual measurement data is input to a recurrent neural network that has been trained to obtain the predicted data at the second time point based on the hidden layer state variables at the second time point, and the predicted data is output. Information processing methods.
9. An acquisition unit for acquiring time series data of actual measurement data; an output unit that inputs the time series data of the actual measurement data acquired into a recurrent neural network that is trained to obtain hidden layer state variables at the second time point under input of external force data at the second time point, hidden layer state variables at a first time point prior to the second time point, and an error between actual measurement data at the first time point and predicted data at the first time point, and to obtain predicted data at the second time point based on the hidden layer state variables at the second time point; and An information processing device comprising:
10. A learning method for a recurrent neural network system including an input layer to which external force data at a second time point is input and a hidden layer, comprising: The hidden layer receives external force data at the second time point from the input layer, hidden layer state variables at a first time point prior to the second time point, and an error between the actual measured data at the first time point and the predicted data at the first time point, and learns using the hidden layer output at the first time point and the error.
Citation Information
Patent Citations
Method and device for learning time-sequential data
JP1997282298A
Cited By
Drainage basin runoff forecasting method fusing attention mechanism and physical recurrent neural network
CN120258259A
