Neural network execution block using fully connected layers

Abstract

Systems and components for use with neural networks. An executive block and a system architecture using the executive block are disclosed. The executive block uses a fully connected stack of layers, with one output being a forecast of a time series and another output being a backcast that can be used to determine residuals from the input to the executive block. The executive block uses a waveform generator subunit that can limit the set of waveforms that can be generated by judiciously selecting parameters. By doing this, the executive block specializes in its function. Systems using the executive block have been shown to outperform the state-of-the-art in providing solutions to time series problems.

Description

[Technical Field]

[0001] This invention relates to computer technology and machine learning. Specifically, it relates to a neural network-based building block architecture and a system suitable for solving time series problems that uses this building block architecture. [Background technology]

[0002] In recent years, the application of deep learning has achieved great success through the invention or adoption of architectures that effectively encode domain-specific inductive biases related to the nature of the problem. Examples include the invention of convolutional neural network (CNN) stacks in the field of image processing, the invention and use of recurrent neural networks (RNNs), attention mechanisms and their combination with CNNs in language modeling, and the invention of prototype networks in few-shot learning.

[0003] In contrast, a significant portion of current research in the machine learning literature related to time series forecasting focuses on applying off-the-shelf deep learning components (primarily RNNs, CNNs, and their combinations) to time series forecasting tasks. This is often done without emphasizing the encoding of time series-specific domain knowledge within the neural architecture. [Overview of the project] [Problems that the invention aims to solve]

[0004] Based on the above, a novel architecture is needed that introduces new concepts to this field, rather than existing solutions. Similarly, these new architectures and systems should preferably surpass the well-known systems and methods currently used to solve time-series problems. [Means for solving the problem]

[0005] This invention provides a system and components for use with neural networks. It discloses an execution block and a system architecture that uses this execution block. The execution block uses a fully connected stack of layers, with one output being a time-series forecast and another output being a backcast that can be used to determine the residuals from the input to the execution block. The execution block uses a waveform generator subunit that can limit the set of waveforms that can be generated by thoughtfully selecting parameters. By doing so, the execution block specializes in its function. Systems using execution blocks have been shown to outperform state-of-the-art technologies in providing solutions to time-series problems.

[0006] In a first aspect, the present invention relates to an execution block used in conjunction with a neural network system, - Includes a stack of fully connected layers of neural network nodes, having outputs received in parallel by a first parallel branch and a second parallel branch, - The first parallel branch described above is It includes a first fully connected layer of a neural network node that receives the output, and a first waveform generator subunit that receives the output of the first fully connected layer, - The second parallel branch mentioned above is, It includes a second fully connected layer of a neural network node that receives the output, and a second waveform generator subunit that receives the output of the second fully connected layer, - The output of the first parallel branch is a composition of the basis functions of the execution block, - The output of the second parallel branch is used to form the residual of the input to the execution block. Provides an execution block.

[0007] In a second aspect, the present invention relates to a neural network system used in time-series forecasting, - Comprising a plurality of base stacks connected in sequence, each base stack includes at least one execution block, and the output of each base stack is added to the cumulative output of the neural network system. - At least one execution block - Includes a stack of fully connected layers of neural network nodes having outputs received in parallel by a first parallel branch and a second parallel branch. - The first parallel branch Includes a first fully connected layer of neural network nodes that receives the output, and a first waveform generator subunit that receives the output of the first fully connected layer. - The second parallel branch Includes a second fully connected layer of neural network nodes that receives the output, and a second waveform generator subunit that receives the output of the second fully connected layer. - The output of the first parallel branch is a composition of the basis functions of the execution block. - The output of the second parallel branch is used to form the residual of the input to the execution block. A neural network system is provided.

[0008] Hereinafter, embodiments of the present invention will be described with reference to the following figures in which the same reference numerals in different figures indicate the same elements.

Brief Description of the Drawings

[0009] [Figure 1] It is a schematic block diagram of an execution block according to one aspect of the present invention. [Figure 2] It is a schematic block diagram of a neural network system architecture according to another aspect of the present invention. [Figure 3] It is a schematic block diagram showing how various aspects of one implementation of the present invention cooperate to form an overall system.

Mode for Carrying Out the Invention

[0010] For further clarification and for the convenience of the reader, the complete citations of the references mentioned in the following explanations can be found in the list at the end of this section. Accordingly, the contents of these references are hereby incorporated by reference in their entirety into this specification.

[0011] To provide context for the present invention, this document considers the univariate point forecast problem. Given a forecast horizon length H, the task is to predict (or forecast) a vector y ∈ R dim(x) of future values given the history x ∈ R H , y = [y1, y2,..., y H . The inventors represent [Number] as the forecast of y. The following metrics are used in this document. [Number]

[0012] Note that sMAPE (symmetric Mean Absolute Percentage Error) and MASE (Mean Absolute Scaled Error) are standard metrics widely used in the forecasting industry. OWA (Overall Weighted Average) is a metric based on the above two other metrics.

[0013] In one aspect, the present invention provides a basic execution block that can be used in an architecture useful for solving the time series / forecast problem described above. Figure 1 shows the basic building block or execution block. This block accepts an input signal x and produces two signals [Number] and

Number

Number

Number

[0014] The basic building block consists internally of two different parts. First, the waveform generator g θ :T H →Y H is a map from the set of points T H ⊂R H in the time domain to the set of points Y H ⊂R in the forecast value domain. The waveform generator is parameterized by a parameter set θ∈Θ. Second, the forward predictor and backward predictor of the parameters of the waveform generator

Number

Number

number

number

[0015] The basic building blocks use basis expansion coefficients based on the input time series window, with forward θ (forecast) and backward θ. b This is a stack of fully connected layers that predicts both (backcasting) and other methods.

[0016] As shown in Figure 1, the execution block 10 has a fully connected stack 20 of multiple neural network layers. This stack 20 has outputs that are received by two parallel branches. The first branch has a fully connected layer 30, and the output of this layer 30 is received by a waveform generator subunit 40. The second branch also has a fully connected layer 50 and a similar waveform generator subunit 60. The output of the first branch is used to determine the residuals of the input to the execution block, and the output of the second branch is used for forecasting.

[0017] Figure 2 shows a block diagram of a system architecture using the execution blocks of Figure 1. The basic building blocks (layers) of this architecture are used to form a stack of layers corresponding to the same output basis function. The weights of fully connected layers are shared within the basis stack. The stack is further chained within the deep neural network.

[0018] Referring to Figure 2, system 100 has multiple base stacks 110, 120, and 130, each generating a forecast. Each of these forecast outputs is added to generate the system output. Within each base stack, there are multiple layers, each being an instance of the execution block shown in Figure 1. As can be seen in the figure, the forecast output of each layer / execution block is added to the stack forecast output. Each backcast output of each layer / execution block is used to determine the residual of the input to that layer / execution block, and this residual is used as the input to the next layer / execution block.

[0019] The architecture in Figure 2 has two residual branches, one flowing over the backcast prediction branch of each block and the other flowing over the forecast branch of each block. The backcast residual branch is interpreted as performing a sequential analysis of the signal, and the gradient backpropagated through the backcast residual branch drives the parameters of each block (Figure 1) to facilitate the tasks performed by subsequent blocks. The forecast residual branch aggregates initially at the stack level and then beyond the stack into the entire forecast (hierarchical aspect). Each stack has the same gθ for each layer. In this case, the interpretation of hierarchical forecast aggregation consists of two elements: (i) each layer flowing down the stream provides a more accurate forecast, and (ii) more importantly, the forecast aggregated at the stack level models the aspect of the forecast encoded in the properties of gθ.

[0020] As can be seen in the diagram, the basic building blocks shown in Figure 1 generate two outputs: a lookback window approximation (backcast) of the block's basis, and a forecast based on the composition of its basis functions. The backcast output of each basic block is used to form the residual of the input to that basic block, and the block approximation of the input to its basis. From Figure 2, the forecast output is added to the output of the basis stack. The residual of the preceding layer is used as the input to the next layer. These basic layers are stacked to form a deep neural network.

[0021] To implement the trend model of the execution block described above, it must be noted that the typical characteristics of a trend are in most cases a monotonic function or at least a slowly changing function. Therefore, to model this type of function, g θ It is sufficient to limit the range of representation to functions that change slowly across the entire forecast window. To mimic this behavior, g θ It is constrained to be a polynomial of small degree p.

number

[0022] Then, the trend forecast will take the following form:

number

number

[0023] To implement a seasonality model using the above architecture, it should be noted that the typical characteristics of seasonality are regular, periodic, and repetitive fluctuations. Therefore, to model seasonality, g θ g is a periodic function θ (t=g θ The function is constrained to belong to the class (ts), where s is the seasonality. A natural choice of basis for modeling the periodic function is a Fourier series.

number

[0024] At this time, the seasonal forecast has the following form:

number

number

[0025] One version of the present invention can also implement an interpretable architecture. This architecture reuses the overall architectural approach shown in the diagram to implement g for each stack θIt can be constructed by adding further structure to it. Performers forecasting time series (TS) often use decomposition of the TS into trend and seasonality, such as that performed by the STL (Seasonal and Trend Decomposition with Loess) method for decomposing time series, developed in 1990, and X13-ARIMA (X13 ARIMA is a software package for seasonal adjustment from the U.S. Census Bureau). In one implementation of the present invention, the trend and seasonality decomposition is designed into a model that makes the stack output more interpretable.

[0026] In one implementation, the overall interpretable architecture consists of two stacks: a trend modeling stack followed by a seasonality modeling stack. Doubly residual stacking, combined with the forecast / backcast principle, results in (i) the trend component being removed from the input window x before being fed into the seasonality analysis stack, and (ii) partial forecasts of trend and seasonality becoming available as separate interpretable outputs. Structurally, each stack consists of multiple blocks connected by residual connections as shown in Figure 1, with each block having its own unlearnable g θ (non-learnable gθ) is shared. The number of blocks is 3 for both trend and seasonality. g θ In addition to sharing weights, we found that sharing all weights across blocks in the stack improved performance on the validation set.

[0027] Figure 3 shows an overall system view of one implementation of the present invention. In this implementation, the basic building block is a multilayer FC network with ReLU nonlinearity. This multilayer FC network has a basis expansion coefficient of θ. f (Forecast) and backward θ b Predict by both (backcasting). The blocks are organized into a stack using the double residual stacking principle. The stack is g θIt can have layers that share a common set of data. Forecasts are aggregated hierarchically. This makes it possible to build very deep neural networks with interpretable outputs.

[0028] For the implementation and testing of this invention, a test subset is derived from a large and diverse 100k series dataset consisting of data frequently encountered in business, finance, and economic forecasting, with sampling frequencies ranging from one hour to one year. In one implementation, the source dataset consists of time series of data by year, quarter, month, and other (weekly, daily, hourly), with a minimum number of observations per dataset of 13 for yearly, 16 for quarterly, 42 for monthly, 80 for weekly, 93 for daily, and 700 for hourly. The time series of the dataset are primarily from economic, finance, demographic, and industrial sectors, but also include data from tourism, trade, labor and wages, real estate, transportation, natural resources, and the environment. The validation and training subsets are obtained by splitting the complete training set at the end-of-period boundary of each time series. The training and validation subsets are used for hyperparameter tuning and reporting of ablation study results. Once the hyperparameters are determined, the model is trained on the complete training set, and a report is made on the test set.

[0029] Tensorflow was used to implement the architecture and train the neural architecture. Network parameters are shared across time periods, resulting in a total of six models, one per period. All time series can be interpreted as separate tasks, thus linking back to multitask learning, which normalizes the neural network by training on multiple tasks to improve generalization. It should be emphasized that models across different time periods reuse the same architecture, and all hyperparameters are fixed to the same values ​​across time periods, with the minor exceptions described later. Since the architecture and hyperparameters can be reused across time periods, this demonstrates that the architecture design of the present invention generalizes well across time series of different natures. The same architecture is successfully trained on periods with 48,000 time series (monthly) and 359 time series (weekly). This is a much more powerful result than those achieved by people who, for example, had to use entirely different architectures handcrafted for different time periods.

[0030] To update the network parameters, each sample is sampled from a fixed-size training batch of 1024 time series (TS) from the same period. Thus, only the neural network corresponding to this period is updated. The batch is constructed by first sampling a period with a probability equal to the proportion of TS in the dataset for that period (for example, a yearly period 6 consisting of 23k TS is sampled with a probability of 0.23). Given a period, 1024 TS IDs from this period are uniformly and randomly selected with permutation. For each selected TS ID, a random forecast point is selected from a historical range of length LHH immediately preceding the last point of the training portion of the TS. LH is a hyperparameter equal to 1.5 for periods with a large number of time series (yearly, monthly, quarterly) and equal to 15 for other periods with a smaller and more modest number of time series (weekly, daily, hourly). Given a sampled forecast point, a single period corresponding to the next point is set as the target forecast window y, and the history of one point of a preceding length of 2H, 3H, ..., 7H is set as the input x to the network. The Adam optimizer is used with default settings and an initial learning rate of 0.001. Neural network training is performed over 30k batches. GPU-based training of one ensemble member takes 30 minutes to 2 hours, depending on the neural network configuration and hardware.

[0031] To test the present invention against known architectures, it should be made clear that the known architectures under test utilize ensemble. Ensemble was also used in the present invention for comparison with these architectures. We chose to construct the ensemble using multiple methods to diversify the individual models. First, the ensemble model is fitted to three different metrics: sMAPE, MASE, and an asymmetric version of sMAPE with ground truth values ​​only in the denominator. The second diversification method is based on including models trained on input windows of different lengths. The ensemble consists of individual models trained on input windows x with six window length options of lengths 2H, 3H, ..., 7H for each period H. Thus, the overall ensemble has a multiscale aspect to it. Finally, a standard bagging procedure was performed by sampling a subset of the time series and a subset of x entries to train each ensemble member. Each individual model is trained on 20% of the time series belonging to its period and uses 75% of the x entries. We used 180 models within the ensemble and cross-validated the ensemble settings on the validation set.

[0032] The performance of two configurations of the present invention—general-purpose (Ours-G), interpretable (Ours-I), and an ensemble of these (Ours-I+G)—was examined. These were compared to four different architectures, each belonging to a different model class. These four architectures are pure ML, pure statistics, ML / TS combination, and DL / TS hybrid. In the table below, the best pure ML is the pure ML architecture, the best statistics is the pure statistical model, the best ML / TS is the machine learning / time series architecture (based on a combination of time series-specific features provided by several statistical time series models and a machine learning layer on top of it), and DL / TS is the deep learning / time series architecture (based on a fusion of a CNN / LSTM stack and a Holt-Winters time series engine with learnable parameters).

[0033] Tables 1 and 2 show the test results. These tables clearly demonstrate that the present invention outperforms all other methods in all subsets of time series, and the OWA gap between our general model and the DL / TS hybrid (0.821 - 0.799 = 0.022) is larger than the gap between the DL / TS hybrid and the second-best architecture (0.838 - 0.821 = 0.017). Our general model is designed specifically for purity. It does not utilize any feature engineering, scaling, or internal structural components that can be considered specific to the time series forecasting problem. [Table 1] [Table 2]

[0034] Therefore, it is found that the present invention works very well in the wide range of time-series forecasting tasks captured by the test dataset described above. Essentially, the present invention forces each execution block into specialization. Instead of having the execution block perform all the functions necessary to forecast consecutive entries in a time series, by limiting the range of waveform functions used by the waveform generator, the execution block specializes in generating smaller subsets / groups of waveforms. Thus, g θ By thoughtfully constraining this, we can select the specialization of each execution block.

[0035] Regarding implementation, the system of the present invention can be implemented on suitable hardware and software as understood by those skilled in the art. In one implementation, it will be apparent that the system can be implemented as a complete software system running on a suitable hardware system. Thus, in such an implementation, each of the stacks and blocks in Figure 3 is a software component in the memory of the hardware system.

[0036] In an alternative implementation, a hardware-based implementation may use one or more processing modules for each block / layer in the architecture. In such an implementation, each layer shown in Figure 2 (and each block in Figure 3) can be implemented as one or more hardware processing modules. A potential problem with such an implementation is that adding a block or layer may require adding hardware processing modules to the system.

[0037] Further implementations can minimize hardware requirements for implementation. In this implementation, a single processing module can be used as a block / layer within the architecture. In such an implementation, data input to block 1 is fed to a single processing module to generate the output of block 1. These output of block 1 is then saved or accumulated as needed. At this time, the relevant output of block 1 (i.e., residual output) is returned to the single processing module along with the input to block 2. The resulting result of block 2 is accumulated along with the result of block 1 as needed, and the relevant result of block 2 (i.e., residual result of block 2) is fed back to the single processing module as input to block 3. The process can then be repeated as needed until a final result is obtained or until a sufficient number of blocks have finished processing the input.

[0038] Another implementation can be a hybrid of the last two implementations described above. This implementation uses a fixed number of processing modules instead of having a single processing module or an infinite number of processing modules. These processing modules are used to process the input as a cascading group. Thus, if there are three processing modules, the first iteration uses these three processing modules to implement blocks 1, 2, and 3. The outputs of these three blocks are accumulated as needed, and the residual output of block 3 is fed back as the input to block 4. Then, in the next iteration, blocks 4, 5, and 6 are implemented using the three processing blocks. This process is then repeated as needed to implement multiple blocks / layers using a fixed number of processing blocks.

[0039] While the above description uses the execution blocks and system of the present invention to solve time-series problems, it will be apparent that these aspects of the invention can be used in other contexts. The execution blocks and system architectures using the execution blocks can be used to solve other problems and in other implementations suitable for neural networks or other machine learning systems / components.

[0040] It will be evident that various aspects of the present invention can be implemented as software modules within an overall software system. Therefore, the present invention can take the form of computer execution instructions that cause various software modules to perform predetermined functions at runtime.

[0041] Embodiments of the present invention may be carried out by a computer processor or a similar device programmed in the form of method steps, or by an electronic system equipped with means for carrying out these steps. Similarly, electronic memory means such as computer diskettes, CD-ROMs, random access memory (RAM), read-only memory (ROM), or similar computer software storage media known in the art may be programmed to carry out such method steps. Furthermore, electronic signals representing these method steps may be transmitted over a communication network.

[0042] Embodiments of the present invention can be implemented in any conventional computer programming language. For example, preferred embodiments can be implemented in a procedural programming language (e.g., "C" or "Go") or an object-oriented language (e.g., "C++", "Java", "PHP", "Python", or "C#"). Other embodiments of the present invention can be implemented as pre-programmed hardware elements, other related components, or combinations of hardware and software components.

[0043] Embodiments can be implemented as computer program products used with computer systems. Such implementations may include a set of computer instructions that can be transmitted to a computer system via a modem or other interface device, such as a communication adapter, which is fixed on a tangible medium (e.g., a diskette, CD-ROM, ROM, or hard disk) or connected to a network via the medium. The medium may be a tangible medium (e.g., optical or telecommunication lines) or a medium that implements wireless technology (e.g., microwave, infrared, or other transmission technology). The set of computer instructions embodies all or part of the functions described herein. Those skilled in the art will understand that such computer instructions can be written in multiple programming languages ​​used with many computer architectures or operating systems. Furthermore, such instructions can be stored in any memory device, such as semiconductor, magnetic, optical, or other memory devices, and transmitted using any communication technology, such as optical, infrared, microwave, or other transmission technology. Such computer program products are expected to be distributed as removable media (e.g., packaged software) accompanied by printed or electronic materials, pre-loaded using a computer system (e.g., on-system ROM or fixed disk), or delivered from a server via a network (e.g., the Internet or the World Wide Web). Naturally, some embodiments of the present invention can be implemented as a combination of both software (e.g., computer program products) and hardware. Yet another embodiment of the present invention can be implemented entirely as hardware or entirely as software (e.g., computer program products).

[0044] Anyone familiar with the present invention may conceive of other structures and embodiments or modifications of the above at this point, but all of these are intended to fall within the scope of the present invention as defined in the following claims.

Claims

1. A non-transient, computer-readable medium storing code for a computer to implement the functionality of an executable block for use with an artificial intelligence neural network system for time-series forecasting, wherein the executable block is - A stack of fully connected layers of a neural network node, the stack having inputs and outputs, the outputs being received in parallel by a first parallel branch and a second parallel branch, including the stack, - The first parallel branch includes a first fully connected layer of a neural network node that receives and processes the output, and a first waveform generator subunit that receives the output of the first fully connected layer. - The second parallel branch includes a second fully connected layer of a neural network node that receives and processes the output, and a second waveform generator subunit that receives the output of the second fully connected layer. - The output of the first parallel branch is a forecast of the execution block based on the input, calculated as a composition of the basis functions of the execution block. - The output of the second parallel branch is an estimate of the execution block for the input and is used to form the residual of the input for the execution block. - The artificial intelligence neural network system is trained over multiple time series so that the artificial intelligence neural network system can be generalized. A non-transient computer-readable medium characterized by the following features.

2. Each of the first and second waveform generator subunits implements a function that maps a set of points in the time domain to a set of points in the forecast value domain. The non-transient computer-readable medium according to claim 1.

3. The execution block is used to forecast the time-series output. The non-transient computer-readable medium according to claim 1.

4. The input to the execution block is an input signal that specifies the history lookback window for the time series values. The non-transient computer-readable medium according to claim 3.

5. The output of the first waveform generator subunit and the second waveform generator subunit is based on a parameter set. The non-transient computer-readable medium according to claim 2.

6. The outputs of the first waveform generator subunit and the second waveform generator subunit each encode an inductive bias for normalizing and constraining the structure of a feasible solution to the time series problem. The non-transient computer-readable medium according to claim 3.

7. The output of the first waveform generator subunit and the second waveform generator subunit is based on a plurality of time-varying waveforms. The non-transient computer-readable medium according to claim 3.

8. The waveform is selected based on a set of parameters selected for each of the first waveform generator subunit and the second waveform generator subunit. The non-transient computer-readable medium according to claim 7.

9. The execution block is used for time-series forecasting in a neural network system. The non-transient computer-readable medium according to claim 2.

10. The non-transient computer-readable medium according to claim 1, wherein the neural network system is used to provide forecasts relating to one or more of the industrial group consisting of economics, finance, demographics and industry, tourism, trade, labor and wages, real estate, transportation, natural resources and environment.

11. A non-transient, computer-readable medium storing code for implementing the functions of an artificial intelligence neural network system used in time-series forecasting, wherein the artificial intelligence neural network system is - Includes multiple base stacks linked in sequence, each base stack containing at least two execution blocks, and the output of each base stack is added to the cumulative output of the neural network system. - Each of at least two execution blocks is: - A stack of fully connected layers of a neural network node, the stack having inputs and outputs, the outputs being received in parallel by a first parallel branch and a second parallel branch, including the stack, - The first parallel branch includes a first fully connected layer of a neural network node that receives and processes the output, and a first waveform generator subunit that receives the output of the first fully connected layer. - The second parallel branch includes a second fully connected layer of a neural network node that receives and processes the output, and a second waveform generator subunit that receives the output of the second fully connected layer. - The output of the first parallel branch is a forecast of the execution block based on the input, calculated as a composition of the basis functions of the execution block. - The output of the second parallel branch is an estimation of the execution block for the input, Used to form the residual of the input to the execution block, - Each of the at least two execution blocks is forcibly specialized by restricting the range of wave functions generated by the corresponding first and second waveform generator subunits. - The artificial intelligence neural network system is trained over multiple time series so that the artificial intelligence neural network system can be generalized. A non-transient computer-readable medium characterized by the following features.

12. Each of the first and second waveform generator subunits implements a function that maps a set of points in the time domain to a set of points in the forecast value domain. The non-transient computer-readable medium according to claim 11.

13. The input to the aforementioned neural network system is an input signal that explicitly defines the historical lookback window of the time-series values. The non-transient computer-readable medium according to claim 12.

14. The output of the first waveform generator subunit and the second waveform generator subunit is based on a parameter set. The non-transient computer-readable medium according to claim 12.

15. The outputs of the first waveform generator subunit and the second waveform generator subunit each encode an inductive bias for normalizing and constraining the structure of a feasible solution to the time series problem. The non-transient computer-readable medium according to claim 12.

16. The output of the first waveform generator subunit and the second waveform generator subunit is based on a plurality of time-varying waveforms. The non-transient computer-readable medium according to claim 12.

17. The waveform is selected based on a set of parameters selected for each of the first waveform generator subunit and the second waveform generator subunit. The non-transient computer-readable medium according to claim 16.

18. The non-transient computer-readable medium according to claim 11, wherein the neural network system is used to provide forecasts relating to one or more of the industrial group consisting of economics, finance, demographics and industry, tourism, trade, labor and wages, real estate, transportation, natural resources and environment.

Images