Prediction systems, training methods and equipment, and computer programs

Abstract

The present invention provides a system for prediction, a training method and apparatus, and a program. [Solution] The training method includes training a predictor so that the M second sequences output from the predictor approach the real sequence; and, after training the predictor, training a refiner model so that the M optimized second sequences output from the refiner model approach the real sequence, while keeping the parameters of the predictor unchanged.

Description

[Technical Field]

[0001] The present invention relates to the field of prediction, and more particularly to prediction of time sequences. [Background technology]

[0002] A time sequence refers to a sequence of data points arranged in chronological order. Typically, the time intervals within a time sequence can be constant, allowing time sequences to be analyzed and processed as discrete data. Time sequence analysis is widely used in various aspects of daily life, such as power systems, transportation systems, finance, weather forecasting, and biomedicine.

[0003] Time sequence analysis is typically divided into four tasks: prediction, interpolation, anomaly detection, and classification. The prediction task aims to predict future values ​​or trends in an input sequence. The interpolation task aims to recover missing data. The anomaly detection task aims to recognize anomalies or anomaly patterns in a sequence. The classification task aims to assign labels or categories (classes) to a given time sequence and is widely used in medical diagnostics.

[0004] In the problem of multivariate time sequence prediction, existing methods can be divided into two types based on their multivariate modeling approach. One is channel-independent (CI) prediction, such as PatchTST, which divides the multivariate sequence into multiple single variables for prediction and models each variable individually. The other is channel-dependent (CD) prediction, such as Informer and FEDformer, which considers the relationships between variables and models multiple variables together.

[0005] Figure 1 illustrates channel-independent and channel-dependent predictions. As shown in Figure 1, if we divide a multivariate sequence into three variables, in this case, channel-dependent prediction models the three variables together (f θ ), and in channel-independent prediction, three variables are modeled separately (f θ1 ,f θ2 ,f θ3 ).

[0006] Channel-independent and channel-dependent predictions each have their own characteristics. Channel-independent predictions consider only a single variable, resulting in simpler models, but they do not consider the relationships between sequences, leading to the loss of some important information. Channel-dependent predictions consider more comprehensive information, but the models become more complex.

[0007] Furthermore, there are methods that combine channel-independent and channel-dependent prediction, aiming to combine the advantages of both. Such a fusion method includes C-LoRA, which performs channel-independent processing in the input layer, models each channel using LoRA, and then performs channel-dependent prediction. However, such fusion methods do not sufficiently improve model performance. [Overview of the project] [Problems that the invention aims to solve]

[0008] The object of the present invention is to provide a prediction system, a training method and equipment, and a computer program. [Means for solving the problem]

[0009] According to one aspect of the present invention, a system for prediction is provided, and such system is A predictor configured to make predictions for M first sequences input and to output M second sequences corresponding to each of the M first sequences, wherein the M first sequences are sequences within a first period and the M second sequences are sequences within a second period following the first period; and The refiner model includes M refiners, and is configured to concatenate each of the M input first sequences with the corresponding sequence from the M second sequences output by the predictor, and then use the M refiners to optimize the sequences belonging to the M second sequences from the M concatenated sequences, and output the M optimized second sequences. M is an integer greater than or equal to 1. The system is trained so that the M optimized second sequences approach the real sequences within the second period.

[0010] According to one preferred implementation, the predictor is a channel-dependent predictor and is configured to make dependent predictions for M first sequences, where the M first sequences are correlated with each other.

[0011] According to one preferred implementation, the predictor consists of M channel-independent predictors, each configured to make a single prediction for a corresponding sequence among the M first sequences.

[0012] According to one preferred implementation, each of the M refiners is configured to prepend one of the M first sequences to the corresponding sequence of the M second sequences.

[0013] According to one preferred embodiment, each of the M refiners is configured to concatenate one of the M first sequences after the corresponding sequence of the M second sequences.

[0014] According to one preferred embodiment, the M first sequences, the M second sequences, and the M optimized second sequences are sequences arranged in chronological order.

[0015] According to one preferred embodiment, the predictor may be a model based on a converter, such as Informer, FEDformer, iTranformer, etc., or may be a linear model such as DLinear, TSMixer, FreTS, etc.

[0016] According to one preferred embodiment, each of the M refiners includes a one-layer linear neural network.

[0017] According to one preferred embodiment, the M first sequences and the M second sequences are normalized before being input into the refiner model, and the refiner model performs inverse normalization on the M optimized second sequences before outputting them.

[0018] According to one preferred embodiment, the training of the system includes the following, that is, first training the predictor, and after training the predictor, training the refiner model while keeping the parameters of the predictor unchanged.

[0019] According to one preferred embodiment, the predictor is trained by minimizing the mean squared error or absolute value error between the M second sequences and the real sequence.

[0020] According to one preferred embodiment, the refiner model is trained by minimizing the mean squared error or absolute value error between the M optimized second sequences and the real sequence.

[0021] According to another aspect of the present invention, a method for training the above-described prediction system is provided, which comprises: training a predictor such that M second sequences output from the predictor approach a real sequence; and after training the predictor and keeping the parameters of the predictor unchanged, training a refiner model such that M optimized second sequences output from M refiners approach the real sequence.

[0022] According to one preferred embodiment, such a method further comprises training the predictor such that the mean squared error or absolute value error between the M second sequences and the real sequence is minimized. [[ID=eleven]]

[0023] [[ID=twelve]] [[ID=thirteen]]According to one preferred embodiment, such a method further comprises training the refiner model such that the mean squared error or absolute value error between the M optimized second sequences and the real sequence is minimized.

[0024] According to one preferred embodiment, such a method further comprises aligning and normalizing the M first sequences and the M second sequences before inputting them into the refiner model, and performing inverse normalization on the M optimized second sequences output by the refiner model before outputting them.

[0025] According to another aspect of the present invention, a device for training the above-described prediction system is provided, which comprises: a first training device used to train a predictor such that M second sequences output from the predictor approach a real sequence; and a second training device used to train a refiner model such that M optimized second sequences output from the refiner model approach the real sequence after training the predictor and keeping the parameters of the predictor unchanged.

[0026] According to one preferred implementation, the first training device is further used to train the predictor such that the mean squared error or absolute error between M second sequences and real sequences is minimized.

[0027] According to one preferred implementation, a second training device is further used to train the refiner model such that the mean squared error or absolute error between the M optimized second sequences and the real sequences is minimized.

[0028] According to yet another aspect of the present invention, a computer-readable storage medium is provided which stores a program, and the program causes the computer to implement the training method described above when executed by a processor.

[0029] According to another aspect of the present invention, corresponding computer program codes and computer program products are further provided. [Effects of the Invention]

[0030] The prediction system, training method, and apparatus according to the present invention can effectively improve the accuracy of predictions for time sequences. [Brief explanation of the drawing]

[0031] [Figure 1] This diagram illustrates channel-dependent and channel-independent predictions. [Figure 2] This figure shows a prediction system according to an embodiment of the present invention. [Figure 3] This diagram shows the operation of the system in Figure 2. [Figure 4] This figure shows the operation of the refiner in Figure 2. [Figure 5] This is a flowchart of a training method according to an embodiment of the present invention. [Figure 6] This is a block diagram showing the configuration of a training device according to an embodiment of the present invention. [Figure 7] This figure intuitively illustrates the improved prediction results achieved by the prediction system according to an embodiment of the present invention, using a curve diagram. [Figure 8] This is a block diagram illustrating an exemplary configuration of a general-purpose personal computer that can implement the method and / or apparatus according to the embodiments of the present invention. Embodiments for carrying out the present invention

[0032] Hereinafter, preferred embodiments for carrying out the present invention will be described in detail with reference to the attached drawings. Note that the following embodiments are merely illustrative and do not limit the present invention.

[0033] To solve the problems of the prior art, the present invention provides a novel fusion system of channel-dependent and channel-independent prediction. The system inputs a historical (past) time sequence into a channel-dependent predictor, establishes one refiner for each channel based on the output time sequence of the channel-dependent predictor, concatenates the output time sequence and the historical time sequence as input to the refiner, and improves the prediction result for each channel by the refiner. The refiner may be, for example, a simple linear neural network with one layer for each channel. As is known, a linear layer, also called a fully connected layer, is a standard layer in a neural network in which each input node is connected to each output node, and each node in the linear layer has a weight vector associated with it, which can be used to calculate the output value.

[0034] The prediction system 100 according to an embodiment of the present invention will be described in detail below, in conjunction with Figures 2 to 4.

[0035] As shown in Figure 2, the system 100 includes a predictor 10 and a refiner model 20. First, M first sequences 11 are input to the predictor 10. The M first sequences 11 are, for example, the sequences in Figure 3.

[0036]

number

[0037]

number

[0038] Next, M first sequences 11 and M second sequences 12 are input to the refiner model 20. M refiners f1, ..., f of the refiner model 20 are input. M Each of these methods concatenates one of the M first sequences 11 with the corresponding sequence from the M second sequences 12 to obtain M concatenated sequences. As can be understood, the second sequences 12 can be concatenated before or after the first sequences 11.

[0039] Next, M refiners f1, ..., f M Based on the M concatenated sequences, the process optimizes the M sequences belonging to the second sequence 12 from among the M concatenated sequences, thereby obtaining M optimized second sequences 12'. The optimized second sequences 12' are, for example, the sequences in Figure 3.

[0040]

number

[0041] According to one preferred implementation, the M first sequences 11 and M second sequences 12 are normalized before being input into the refiner model 20, that is, the data is mapped to a range of 0 to 1, and correspondingly, the refiner model 20 is denormalized before outputting the M optimized second sequences 12'. In this way, the adverse effects of mismatches in data distribution can be reduced, the generalization performance of the system can be improved, and training can be accelerated.

[0042] The predictor 10 may be a transformer-based model, such as Informer, FEDformer, or iTranformer, or a linear model such as DLinear, TSMixer, or FreTS.

[0043] Furthermore, as shown in Figure 4, each of the refiners 1 through M may be implemented by a single-layer linear neural network for the sequence after the M channels have been combined. However, as can be understood, the implementation method of refiners 1 through M is not limited to this, and they may also be implemented using convolutional neural networks, recurrent neural networks, converters, etc.

[0044] Furthermore, the training of system 100 may be performed by training the predictor 10 and the refiner model 20 simultaneously.

[0045] According to one preferred implementation, training of system 100 is divided into two stages. First, the predictor 10 is trained so that its prediction of future sequences of the training set data approaches the actual future sequences as closely as possible. Next, after training the predictor 10 and with the parameters of the predictor 10 fixed, the refiner model 20 is trained so that its prediction of future sequences of the training set data approaches the actual future sequences as closely as possible.

[0046] To be understandable, the training of the refiner model 20 includes training each of the refiners 1 to M, and the parameters of the trained refiners 1 to M are the same as or different from each other.

[0047] In the case of two-stage training, assuming that the predictor 10 is a channel-dependent predictor, in this case, for example, the objective function can be set to the mean squared error loss function l' of Equation 1) as follows.

[0048]

Number

[0049] The objective function of the refiner model 20 is set as the mean squared error loss function l of Equation 2) as follows.

[0050]

Number

[0051]

Number

[0052]

number

[0053] In the two-stage training, first the predictor 10 is trained using the loss function l', and then, after the training of the predictor 10 is complete and the parameters of the predictor 10 are kept unchanged, the refiner model 20 is trained using the loss function l.

[0054] It should be noted that constructing the loss functions l' and l as shown in equations 1) and 2) above is merely illustrative, and the present invention is not limited to these. Any other suitable method may be adopted, that is, as long as the prediction results of the predictor 10 and the refiner model 20 are made to approach real values ​​as closely as possible.

[0055] When the predictor 10 and the refiner model 20 are trained simultaneously, the total loss function L may be the sum of equations 1) and 2) above, for example, as shown in equation 3). L = αl' + βl (3) Of these, α and β represent weights and may be set according to actual needs.

[0056] To make it clear, the loss function l' in equation 1) above is constructed based on channel-dependent prediction, but the loss function l' may similarly be constructed based on channel-independent prediction. For the sake of brevity, we will not go into detail here about constructing the loss function based on channel-independent prediction.

[0057] Figure 5 is a flowchart of the training method 500 according to the present invention. The method 500 in Figure 5 will be described below in conjunction with the system 100 in Figure 2.

[0058] First, in step 501, the predictor is trained so that the M second sequences output from the predictor approach the real sequences. For example, in this embodiment, the predictor 10 is trained using a loss function l' as shown in equation 1) so that the prediction results output from the predictor 10 approach the real sequences in the dataset.

[0059] Next, in step 502, after training the predictor and keeping the predictor parameters unchanged, the refiner model is trained so that the M optimized second sequences output from the refiner model approach the real sequences. For example, in this embodiment, the refiner model 20 is trained using a loss function l as shown in equation 2) so that the prediction results output from the refiner model 20 approach the real sequences in the dataset.

[0060] The above method may be implemented entirely by a computer-executable program, or partially or entirely by hardware and / or firmware. When implemented by hardware and / or firmware, or when a computer-executable program is loaded into a hardware device capable of executing the program, the training equipment described below can be realized. Below, an overview of such equipment is provided without repeating some of the above details, but although such equipment can perform the method described above, the method does not necessarily have to employ or be performed by the components of the equipment described.

[0061] Figure 6 is a block diagram showing a training device 600 according to the present invention. The device 600 includes a first training device 601 and a second training device 602. The first training device 601 is used to train a predictor so that the M second sequences output from the predictor approach the real sequence. The second training device 602 is used, after the predictor has been trained and while keeping the predictor parameters constant, to train a refiner model so that the M optimized second sequences output from the refiner model approach the real sequence.

[0062] According to one preferred implementation, the first training device 601 is further used to train the predictor such that the mean squared error or absolute error between M second sequences and real sequences is minimized.

[0063] According to one preferred implementation, the second training device 602 is further used to train the refiner model such that the mean squared error or absolute error between the M optimized second sequences and the real sequences is minimized.

[0064] The training equipment 600 shown in Figure 6 corresponds to the training method 500 shown in Figure 5. Therefore, the details of each device in the training equipment 600 are given in the explanation of the training method 500 shown in Figure 5, and a detailed explanation is omitted here.

[0065] <Experiment> The following table compares the prediction results of the prediction system according to the present invention with those of conventional models. The experiments were based on the datasets ETTh1, ETTm1, and Electricity, and the models included the linear model DLinear and the transformer structure-based deep models iTransformer, FEDformer, and Informer. Time sequence predictions were evaluated using mean squared error (MSE) and absolute error (MAE), with lower values ​​indicating better performance.

[0066] [Table 1] As shown in Table 1, the prediction system according to the present invention can effectively improve the accuracy of predictions.

[0067] Figure 7 is a diagram that intuitively illustrates, in the form of a curve, that the prediction results output from the prediction system according to the present invention are matched with real sequences. As shown in Figure 7, the optimized prediction results (curve c) are closer to the real values ​​of the sequences (curve a) than the prediction results of the conventional model (curve b).

[0068] Therefore, the prediction system 100 and training method 500 of the present invention, that is, by combining a channel-dependent strategy and a channel-independent strategy within a single network, can combine the advantages of each of these two channel strategies to improve the accuracy of time sequence prediction.

[0069] Furthermore, experiments have verified the following: the refiner according to the embodiment of the present invention can improve the performance of multiple basic predictive models on any real-world dataset.

[0070] Each module or unit in the above-described apparatus or device can be configured as software, firmware, hardware, or a combination thereof. Specific means or methods that may be used in configuration are well known to those skilled in the art and will not be described in detail here. When implemented by software or firmware, the software is installed from a storage medium or network to a computer with a dedicated hardware structure (for example, the general-purpose computer 800 shown in Figure 8), and the computer can perform various functions when various programs are installed.

[0071] Figure 8 is a block diagram showing an exemplary structure of a general-purpose personal computer that can implement a method and / or apparatus according to an embodiment of the present invention. As shown in Figure 8, the CPU 801 executes various processes based on a program stored in the ROM 802 or a program loaded into the RAM 803 from the storage device 808. The RAM 803 also stores data necessary for the CPU 801 to execute various processes, as needed. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output interface 805 is also connected to the bus 804.

[0072] Furthermore, the input / output interface 805 is connected to the following components: an input device 806 including a keyboard, an output device 807 including a display such as a liquid crystal display (LCD) and a speaker, a storage device 808 including a hard disk, and a communication device 809 including a network interface card, such as a LAN card or modem. The communication device 809 performs communication processing via a network such as the Internet or a LAN. A drive 810 may also be connected to the input / output interface 805 as needed. A removable medium 811, such as a semiconductor memory, can be set in the drive 810 as needed, allowing a computer program read from it to be installed in the storage device 808.

[0073] When the above-described series of processes are implemented by software, the processes constituting the software can be installed from a network, such as the Internet, or from a storage medium, such as a removable medium 811. Furthermore, the present invention also provides a program product including a machine-readable instruction code. When such an instruction code is read and executed by a machine, it can perform the methods described in the embodiments of the present invention described above. Accordingly, various storage media that carry such program products, such as magnetic disks (including floppy disks®), optical disks (including CD-ROMs and DVDs), magneto-optical disks (including MDs®), and semiconductor memory devices, are also included in the present invention.

[0074] Furthermore, each operation (process) in the above-described method can also be implemented as a computer-executable program stored on various machine-readable storage media.

[0075] Furthermore, the above embodiments and other details are disclosed as follows:

[0076] (Note 1) A system for prediction, A predictor configured to make predictions for M first sequences input and to output M second sequences corresponding to each of the M first sequences, wherein the M first sequences are sequences within a first period and the M second sequences are sequences within a second period following the first period; and The refiner model includes M refiners, and is configured to concatenate each of the M input first sequences with the corresponding sequence from the M second sequences output from the predictor, and to optimize the sequences belonging to the M second sequences from the M concatenated sequences using the M refiners, based on each of the M sequences, and to output the M optimized second sequences. Among them, M is an integer greater than or equal to 1, Eventually, the system is trained so that the M optimized second sequences approach the real sequences within the second period.

[0077] (Note 2) The system described in Appendix 1, The predictor is a channel-dependent predictor, configured to make dependent predictions for M first sequences, where the M first sequences are correlated with each other.

[0078] (Note 3) The system described in Appendix 1, The predictor consists of M channel-independent predictors, each configured to make a single prediction for the corresponding sequence among the M first sequences.

[0079] (Note 4) A system according to any one of the appendices 1 to 3, Each of the M refiners is configured to sequence one of the M first sequences with the corresponding sequence from the M second sequences.

[0080] (Note 5) A system according to any one of the appendices 1 to 3, Each of the M refiners is configured to sequence one of the M first sequences with the corresponding sequence from the M second sequences.

[0081] (Note 6) A system according to any one of the appendices 1 to 3, The M first sequences, M second sequences, and the M optimized second sequences are sequences arranged in chronological order.

[0082] (Note 7) A system according to any one of the appendices 1 to 3, The predictor is a transformer-based model such as Informer, FEDformer, or iTranformer, or a linear model such as DLinear, TSMixer, or FreTS.

[0083] (Note 8) A system according to any one of the appendices 1 to 3, Each of the M refiners contains a single-layer linear neural network.

[0084] (Note 9) A system according to any one of the appendices 1 to 3, The M first sequences and M second sequences are normalized before being input to the refiner model, and the refiner model then denormalizes the M optimized second sequences before outputting them.

[0085] (Note 10) A system according to any one of the appendices 1 to 3, The system training involves first training a predictor, and then, after training the predictor, training a refiner model while keeping the predictor's parameters constant.

[0086] (Note 11) The system is as described in Appendix 10, This method trains a predictor by minimizing the mean squared error or absolute error between M second sequences and the real sequence.

[0087] (Note 12) The system is as described in Appendix 10, This method trains M refiners by minimizing the mean squared error or absolute error between the M optimized second sequences and the real sequences.

[0088] (Note 13) A method for training a system of any one of the items in Appendix 1 to 8, The predictor is trained so that the M second sequences output from the predictor approach the real sequence; and This method involves training a refiner model after training a predictor, while keeping the predictor's parameters constant, such that the M optimized second sequences output from the refiner model approach the real sequences.

[0089] (Note 14) The method described in Appendix 13, The method further includes training the predictor such that the mean squared error or absolute error between M second sequences and the real sequence is minimized.

[0090] (Note 15) The method described in Appendix 13, The refiner model further includes training it to minimize the mean squared error or absolute error between the M optimized second sequences and the real sequences.

[0091] (Note 16) Any method of any one of the appendices 13 to 15, This further includes performing a normalization process on M first sequences and M second sequences before inputting them into a refiner model, and performing an inverse normalization process on the M optimized second sequences before outputting them from the refiner model.

[0092] (Note 17) Equipment for training any one of the systems specified in Appendix 1 to 8, A first training device used to train the predictor so that the M second sequences output from the predictor approach the real sequence; and A refiner model, after training a predictor and keeping the predictor parameters constant, includes a second training device used to train the refiner model such that the M optimized second sequences output from the refiner model approach the real sequences.

[0093] (Note 18) The equipment described in Appendix 17, The first training device is used to train the predictor so that the mean squared error or absolute error between M second sequences and the real sequence is minimized.

[0094] (Note 19) The equipment described in Appendix 17, The second training device is used to train the refiner model so that the mean squared error or absolute error between the M optimized second sequences and the real sequences is minimized.

[0095] (Note 20) A computer-readable storage medium in which a program is stored, A program, when executed by a processor, causes the computer to perform any one of the methods described in appendices 13 to 16.

[0096] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and any modification to the present invention that does not deviate from the spirit of the invention falls within the technical scope of the present invention.

Claims

1. A system for prediction, A predictor configured to make predictions for M first sequences input and to output M second sequences corresponding to each of the M first sequences, wherein the M first sequences are sequences within a first period and the M second sequences are sequences within a second period following the first period; and Includes a refiner model containing M refiners, The aforementioned refiner model is Each of the M first sequences input is concatenated with the corresponding sequence from the M second sequences output from the predictor. Based on each of the M sequences after the splicing, the M refiners are used to optimize the sequences belonging to the M second sequences among the M sequences after the splicing, and Output M optimized second sequences. It is configured in such a way, M is an integer greater than or equal to 1, The system is trained so that the M optimized second sequences approach the real sequences within the second period.

2. The system according to claim 1, The predictor is a channel-dependent predictor and is configured to perform dependent predictions for the M first sequences. The aforementioned M first sequences are correlated with each other, forming a system.

3. The system according to claim 1, The system comprises M channel-independent predictors, each configured to make independent predictions for corresponding sequences in the M first sequences.

4. A system according to any one of claims 1 to 3, A system in which each of the M refiners is configured to sequence one sequence from the M first sequences before or after a corresponding sequence from the M second sequences.

5. A system according to any one of claims 1 to 3, Each of the M refiners is a system containing a single-layer linear neural network.

6. A system according to any one of claims 1 to 3, The training of the aforementioned system is Train the predictor; and A system comprising training the refiner model after training the predictor, while keeping the parameters of the predictor unchanged.

7. The system according to claim 6, The predictor is trained by minimizing the mean squared error or absolute error between the M second sequences and the real sequence. A system for training the refiner model by minimizing the mean squared error or absolute error between the M optimized second sequences and the real sequences.

8. A method for training the system according to any one of claims 1 to 3, The predictor is trained such that the M second sequences output from the predictor approach the real sequence; and A method comprising, after training the predictor, training the refiner model such that the M optimized second sequences output from the refiner model approach the real sequences, while keeping the parameters of the predictor constant.

9. A device for training the system described in any one of claims 1 to 3, A first training device used to train the predictor such that the M second sequences output from the predictor approach the real sequence; and An apparatus including, after training the predictor, a second training device used to train the refiner model such that the M optimized second sequences output from the refiner model approach the real sequences, while keeping the parameters of the predictor constant.

10. A program for causing a computer to perform the method described in claim 8.

Images