Real-time calculation of Temporal Convolutional Network

Abstract

The Real-Time Temporal Convolutional Network (RT-TCN) algorithm reuses the output of previous convolutional operations in all layers of the network to minimize the computational requirements and memory footprint of the TCN during real-time evaluation. Additionally, a TCN trained via a fixed-window view can be continuously executed in real-time using RT-TCN, where the TCN is trained using a fixed-time concatenation of the input time series.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to machine learning systems and methods. More specifically, the present invention relates to systems and methods for computing temporal convolutions in time series data in real time, where temporal convolutional networks need to be continually evaluated as new data becomes available. Background Art

[0002] The following background information may present examples of specific aspects of the prior art (e.g., not limited to methods, facts, or common knowledge) which, while expected to help further educate the reader about other aspects of the prior art, should not be construed as limiting the present invention or any embodiment thereof to anything stated, implied, or inferred therefrom.

[0003] Many current solutions to problems that operate on time-series inputs, such as human activity recognition and heartbeat detection, now use deep neural networks (DNNs). Among different DNN architectures, temporal convolutional networks (TCNs) have demonstrated outstanding results on both synthetic and real-world datasets. In a TCN, the convolution operation is applied along the time dimension, convolving data from many different time steps together. By constructing layers of such temporal convolutions, a single output can incorporate the long-term history of the input data. This input history is called the receptive field of the TCN.

[0004] If the TCN is being executed continuously in real time on a device with sufficient computational bandwidth and memory, then the last input comprising the TCN's receptive field can simply be stored in memory, and some individual convolutions in the TCN can be redundantly recomputed to compute the latest output. However, when resources are limited, executing a trained TCN in real time can be challenging.

[0005] In summary, it is clear that there is a need for an algorithm that can address some or all of the shortcomings of traditional methods. Summary of the Invention

[0006] An embodiment of the present invention provides a computerized method for performing deep convolution on time series data in a temporal convolutional network in real time, comprising inputting time series data of a first fixed time window; predicting a first output through a trained deep convolutional network; inputting time series data of a second fixed time window; predicting a second output through a trained deep convolutional network, wherein the convolution operation required to provide the first output and the second output is not repeated for the prediction of the second output or for subsequent predicted outputs.

[0007] Embodiments of the present invention also provide a computerized method for performing depth convolution on time series data in a temporal convolutional network, including inputting time series data of a first fixed time window; performing a ready convolution operation once all inputs required for performing the ready convolution operation are available; retaining in a memory inputs and intermediate values required for performing future convolution operations; predicting a first output through a trained depth convolutional network; inputting time series data of a second fixed time window; predicting a second output through the trained depth convolutional network, wherein the convolution operations required for the second output retained in the memory are reused instead of re - performing the convolution operations.

[0008] Embodiments of the present invention also provide a non - transitory computer - readable storage medium storing an executable program, wherein the program instructs one or more processors to perform actions including inputting time series data of a first fixed time window; predicting a first output through a trained depth convolutional network; inputting time series data of a second fixed time window; predicting a second output through the trained depth convolutional network, wherein the convolution operations required for providing the first output and the second output are not repeated for the prediction of the second output or for subsequent predicted outputs.

[0009] These and other features, aspects, and advantages of the present invention will be better understood with reference to the following drawings, description, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Some embodiments of the present invention are illustrated by way of example and are not limited by the figures of the drawings, in which like reference numerals may indicate similar elements.

[0011] Figure 1 An exemplary temporal convolutional network with three layers and a receptive field size T = 15 is illustrated, where all layers apply 1 - D temporal convolution with a kernel size of 3. The dilation rate of layer 1 is 1 and the stride is 2, the network dilation rate of layer 2 is 1 and the stride is 1, and the network dilation rate of layer 3 is 2 and the stride is 1. Wherein, because the stride of layer 1 is 2, the temporal dilation rate of layer 2 is 2, and the temporal dilation rate of layer 3 is 4. Wherein the input, intermediate convolutional outputs, and convolution pairs marked by dashed lines contribute to the output at the current time t - 2, and the black part contributes to the output at time t;

[0012] Figure 2 Illustrated is Figure 1 the fixed - window view of the temporal convolutional network when calculating a single output;

[0013] Figure 3Illustrated are the separate convolutional operations required to process a fixed window to generate a set of two output values for each output, where the set of these output values can be used directly or further processed by the system, where for output n the convolution illustrated by the bold solid line is required, and for output n+1 the convolution illustrated by the bold dashed line is required;

[0014] Figure 4 Illustrated is a temporal convolutional network where output n+1 needs to reuse a portion of output n-1 to prevent redundant re-evaluation of the convolution, where for output n the convolution illustrated by the bold solid line is required, and for output n+1 the convolution illustrated by the bold dotted-dashed line is required; and

[0015] Figure 5 Illustrated is an enumeration of the fixed-window outputs that use the input or the output of the convolutional operation within the network, where the arrows indicate which values are used in a single convolutional operation to generate the values for the next layer, and follow the same interleaving as in Figure 3 and Figure 4 where the previous calculations required to compute the values are retained in memory, and these previous calculations together with the currently computed values are marked in bold.

[0016] Unless otherwise indicated, the illustrations in the various figures are not necessarily drawn to scale.

[0017] Now, the present invention and its various embodiments can be better understood by turning to the following detailed description in which the illustrated embodiments are described. It should be clearly understood that the illustrated embodiments are set forth by way of example only and not as a limitation of the invention as finally defined in the claims. Detailed Description

[0018] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well as the singular forms unless the context clearly indicates otherwise. It will also be understood that when used in this specification, the terms "comprises" and / or "comprising" specify the presence of the stated features, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components and / or groups thereof.

[0019] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0020] In describing the present invention, it will be understood that numerous techniques and steps are disclosed. Each of these techniques and steps has its respective benefits, and each can also be used in combination with one or more of the other disclosed techniques, or in some cases, with all of the other disclosed techniques. Thus, for clarity, the description will avoid repeating each possible combination of the individual steps in an unnecessary manner. However, the specification and claims should be read with the understanding that such combinations are fully within the scope of the present invention and the claims.

[0021] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present invention may be practiced without these specific details.

[0022] This disclosure is considered to be illustrative of the present invention and is not intended to limit the present invention to the specific embodiments illustrated by the following figures or description.

[0023] "Computer" or "computing device" can refer to one or more apparatuses and / or one or more systems that are capable of accepting structured input, processing the structured input according to prescribed rules, and producing the results of the processing as output. Examples of a computer or computing device can include: a computer; a fixed and / or portable computer; a computer having a single processor, multiple processors, or multi-core processors that can operate in parallel and / or not in parallel; a general-purpose computer; a supercomputer; a mainframe; a super-minicomputer; a minicomputer; a workstation; a microcomputer; a server; a client; an interactive television; a web appliance; a telecommunications device having Internet access; a hybrid combination of a computer and an interactive television; a portable computer; a tablet personal computer (PC); a personal digital assistant (PDA); a portable telephone; special-purpose hardware for emulating a computer and / or software (such as, for example, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), one chip, multiple chips, a system-on-a-chip, or a chip set); a data acquisition device; an optical computer; a quantum computer; a biometric computer; and an apparatus that generally can accept data, process the data according to one or more stored software programs, generate results, and generally includes input, output, storage, arithmetic, logic, and control units.

[0024] "Software" or "application" can refer to the prescribed rules for operating a computer. Examples of software or an application can include segments of code in one or more computer-readable languages; graphical and / or textual instructions; applets; pre-compiled code; interpreted code; compiled code; and computer programs.

[0025] The example embodiments described herein may be implemented in an operating environment (e.g., software) including computer-executable instructions installed on a computer, in hardware, or in a combination of software and hardware. The computer-executable instructions may be written in a computer programming language or may be implemented in firmware logic. If written in a programming language conforming to recognized standards, such instructions may be executed on a variety of hardware platforms and for interfaces to a variety of operating systems. Although not limited thereto, computer software program code for performing operations in accordance with aspects of the present invention may be written in any combination of one or more suitable programming languages, including object-oriented programming languages and / or traditional procedural programming languages, and / or programming languages such as, for example, HyperText Markup Language (HTML), Dynamic HTML, Extensible Markup Language (XML), Extensible Stylesheet Language (XSL), Document Style Semantics and Specification Language (DSSSL), Cascading Style Sheets (CSS), Synchronized Multimedia Integration Language (SMIL), Wireless Markup Language (WML), Java.TM., Jini.TM., C, C++, Smalltalk, Python, Perl, UNIX Shell, Visual Basic or Visual Basic Script, Virtual Reality Markup Language (VRML), ColdFusion.TM. or other compilers, assemblers, interpreters or other computer languages or platforms.

[0026] Computer program code for performing operations in accordance with aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++ and traditional procedural programming languages such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider). The program code may also be distributed among a plurality of computing units, where each unit processes a portion of the total computation.

[0027] Aspects of the present invention will now be described with reference to the flowchart and / or block diagram of a method, apparatus (system), and computer program product according to an embodiment of the present invention. It will be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed via the computer or other programmable data processing apparatus create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block Figure 1 diagram.

[0028] These computer program instructions may also be stored in a computer-readable medium, which can direct a computer, other programmable data processing apparatus, or other devices to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in one or more blocks of the flowchart and / or block Figure 1 diagram.

[0029] In addition, while process steps, method steps, algorithms, etc. may be described in a sequential order, such processes, methods, and algorithms may be configured to work in an alternative order. In other words, any sequence or order of steps that can be described does not necessarily indicate a requirement to execute the steps in that order. The steps of the processes described herein can be executed in any practical order. Additionally, some steps may be executed simultaneously.

[0030] It will be apparent that the various methods and algorithms described herein can be implemented by, for example, appropriately programmed general purpose computers and computing devices. Generally, a processor (e.g., a microprocessor) will receive instructions from a memory or similar device and execute those instructions, thereby performing the processes defined by those instructions. Additionally, various known media can be used to store and transmit programs that implement these methods and algorithms.

[0031] As used herein, the term "computer-readable medium" refers to any medium that participates in providing data (e.g., instructions) that can be read by a computer, a processor, or a similar device. Such a medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks and other permanent storage. Volatile media includes dynamic random access memory (DRAM), which typically constitutes main memory. Transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that make up a system bus coupled to a processor. Transmission media can include or convey acoustic waves, light waves, and electromagnetic radiation (such as acoustic, light, and electromagnetic radiation generated during radio frequency (RF) and infrared (IR) data communications). Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASHEEPROM, any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read.

[0032] Various forms of computer-readable media may be involved in carrying a sequence of instructions to a processor. For example, a sequence of instructions (i) may be delivered from RAM to a processor, (ii) may be carried on a wireless transmission medium, and / or (iii) may be formatted according to various formats, standards, or protocols (such as Bluetooth, TDMA, CDMA, 3G, 4G, 5G, etc.).

[0033] Embodiments of the present invention may include apparatus for performing the operations disclosed herein. The apparatus may be specially constructed for the desired purpose or the apparatus may comprise a general-purpose device selectively activated or reconfigured by a program stored in the device.

[0034] Unless otherwise specifically stated, and as will be apparent from the following description and claims, it should be recognized that throughout the specification description, terms such as "processing," "computing," "calculating," "determining," etc. refer to the actions and / or processes of a computer and computing system or similar electronic computing device that manipulate and / or transform data represented as physical (such as electronic) quantities within the registers and / or memories of the computing system into other data similarly represented as physical quantities within the memories, registers, or other such information storage devices, transmission, or display devices of the computing system.

[0035] In a similar manner, the term "processor" can refer to any device or portion of a device that processes electronic data from registers and / or memory to transform that electronic data into other electronic data that can be stored in registers and / or memory or can be transmitted to an external device to cause a physical change or actuation of the external device.

[0036] An algorithm is here and generally considered to be a self-consistent sequence of actions or operations that leads to a desired result. These include physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated. It has been shown that, sometimes, mainly for reasons of common usage, it is convenient to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc. However, it should be understood that all of these and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to these quantities.

[0037] Broadly speaking, embodiments of the present invention provide a real-time (RT) temporal convolutional network (TCN) algorithm that reuses the output of previous convolutional operations in all layers of the network to minimize the computational requirements and memory footprint of the TCN during real-time evaluation. Additionally, an RT-TCN can be continuously executed in real-time via a fixed-window view trained TCN, where the TCN is trained using a fixed-time splicing of the input time series.

[0038] The RT-TCN operates using the following two principles: (1) Execute the convolutional operation once all the inputs required for the execution of the convolutional operation are available. (2) At most, the dilated kernel width is required at any given layer to compute the next convolution.

[0039] The RT-TCN consists of two parts: (1) Algorithm 1 runs once to generate a buffer to store the input and intermediate data. (2) Then the generated buffer is supplied to Algorithm 2 and continuously run. Given the latest input, Algorithm 2 performs the required individual convolutions in all layers, updates the required buffer, and returns the network output if available. It should be noted that if (1) the TCN is not filled and the number of inputs received before the current time is less than the receptive field size; or (2) the output rate is less than the input rate, as Figure 1 in the example, then the network output may not be available at a certain time step.

[0040] An RT-TCN algorithm for computing the TCN output is provided that buffers the previous convolutional outputs to ensure that no individual convolution is redundantly computed in this problem setting. Specifically, in the exemplary TCN shown in Figure 1 the RT-TCN will only compute y t-2 versus yt The convolution shared among them is performed once. In addition, for some TCN architectures, RT-TCN also reduces the memory footprint compared to the direct method of buffering the last T inputs, where T is the size of the receptive field of the TCN. On devices with limited computational bandwidth and memory, RT-TCN allows the execution of some TCN architectures that would otherwise be impossible.

[0041] Although the TCN is computed continuously in real time, it can be more convenient to train the TCN from pre-recorded data by backpropagating the error over a fixed window of time series data. In this fixed window view of the TCN, each fixed window can generate only one or a few outputs. For example, in a reinforcement learning (RL) setting, where the TCN output can represent the actions being taken in a system, it can be convenient to backpropagate the error for one action / output at a time. When the TCN is trained via the fixed window view of the input time series (as Figure 2 shown), and then executed continuously in real time at a specified output rate (as Figure 1 shown), the network architecture parameters in this continuous view can be different from the fixed window view. The transition from the fixed window view to the continuous view of the TCN provides a simple session method that allows the TCN to be trained in the fixed window representation and then executed continuously via RT-TCN.

[0042] Generally, convolutional networks have been applied in the field of image processing, where 2D convolutions are applied across the width and height dimensions of an image. The TCN is a 1D convolutional network where the input data is supplied as a time series. The TCN captures the history of the input, where the input at each time step can include different channels (i.e., features) and performs 1D convolutions on this input data to generate an output. Similar to general convolutional networks, the TCN also includes many different convolutional layers, and the depth of the TCN is defined by the number of layers (n) inside it. The receptive field includes the history of the input that contributes to a single output in the TCN. For the Figure 1 example presented in Figure 1 , the depth of the network is 3, and the receptive field size is 15.

[0043] Aspects of the present invention contemplate a TCN composed of causal convolutions. A convolution operation is considered causal if its output is generated by convolving the input data from the current time step or earlier time steps. Causality ensures that the TCN does not depend on future data, and thus the TCN is suitable for real-time execution. For the Figure 1 example presented in Figure 1 , the fact that the last input in the convolution is aligned with the output along the x-axis (time dimension) implies causality.

[0044] For any given TCN layer, the convolution operation is defined via the following parameters:

[0045] (1) Convolution kernel size (k s ). The kernel size is the number of time steps that are convolved together in a single convolution operation. In Figure 1 , the kernel size of all layers is 3.

[0046] (2) Convolution filters (n f ) - The number of convolution filters represents the number of independent convolution operations applied on the same input in order to generate multiple output channels at a single time step. These output channels are then supplied as input channels to the next layer. At the output layer, the number of filters represents the number of channels in the output at each time step. The number of input channels (c) and the number of convolution filters are not represented in Figure 1 , but are used by RT-TCN to describe the dimensions of the input and intermediate buffers.

[0047] (3) Network dilation rate (d). Network dilation is performed by feeding the history of the input to a single convolution that is longer than the kernel size by skipping intermittent inputs. Increasing the network dilation at each layer allows increasing the receptive field size without increasing the depth of the network or the kernel size at each layer. This feature allows increasing the historical influence of the input on the output and is useful for domains that require a longer history to solve the task. In Figure 1 , for three layers, the network dilation rates are 1, 1, and 2 respectively.

[0048] (4) Convolution stride (s). The convolution kernels in a single layer are repeatedly applied at the intervals of the convolution stride. In Figure 1 , for three layers, the strides are 2, 1, and 1 respectively. The output rate of the TCN is reduced by a factor where s i is the stride size of the i-th layer. For example, for the TCN in Figure 1 , the output rate is reduced by a factor of 2 X 1 X 1 = 2. Additionally, the downsampling caused by the convolution stride affects the temporal dilation of the layers that depend on the output. In Figure 1 , the network dilation rates 1, 1, and 2 are equivalent to the temporal dilation rates 1, 2, and 4 respectively. Generally, for a given i-th layer, if the stride s i ≠ 1, then the network dilation rate d i = 1, and vice versa. This constraint ensures that all inputs at the current layer contribute to the output.

[0049] (5) Padding (p). In an un-padded TCN, the first prediction will not be available until the number of input samples is equal to the receptive field of the TCN. Alternatively, padding the convolutional layer involves filling the edges of the input with default values (usually zero) so that there is enough input available to make the first prediction faster. For example, if Figure 1If the first layer of the TCN uses a padding size of p = 14, then after providing the first sample, the prediction for the entire network will be available.

[0050] (6) Activation function (g). The activation function of a layer is an element-wise non-linear function applied to the result of a regular convolutional operation. As with a regular feed-forward neural network, using an activation function on intermediate layers allows the network to model complex non-linear relationships. The choice of activation function in a TCN is typically the same as that used in a feed-forward network; for example, the rectified linear unit (ReLU), sigmoid, tanh, or exponential linear unit (ELU).

[0051] As Figure 3 shown, once output n has been generated, 11 of the 18 individual convolutional operations required to generate output n+1 have already been computed. By efficiently keeping track of previous inputs and convolutional outputs, the algorithm presented in aspects of the present invention can eliminate redundant computations.

[0052] The three main design goals of this algorithm are (1) a particular convolutional operation using the same input is not performed more than once across multiple input fixed windows; (2) a convolutional operation is performed once all the inputs required for the operation are available; and (3) the inputs and intermediate values required for future convolutional operations are retained in memory.

[0053] It should be noted that in some cases, the convolutional operations required to finalize the output of a particular window may have been performed many windows prior, not just in the previous time window. For example, in Figure 4 it is demonstrated a case where the output from the network needs to be computed every 4 samples instead of 8 samples. In this setting, the convolutional operations required for output n+1 were performed during the computation of output n-1. Output n+1 needs to reuse a portion of output n-1 to prevent redundant re-evaluation of the convolution. In Figure 4 for output n the convolution illustrated by the bold solid line is required, and for output n+1 the convolution illustrated by the bold dotted line is required.

[0054] Furthermore, there are even more complex cases that require intelligent handling of previous inputs and outputs. Figure 5 illustrates the same example as before, except that the network now needs to (instead of as Figure 4Generate new outputs as in [4] for every 4 samples. This figure enumerates whether the input or the output of the convolutional operation is required for computing the output of a specific window, and which of these previous values need to be retained in memory. Since the stride at each layer is two, as the values required as input to a single convolutional operation at the next layer move from the input layer to the output layer, they become increasingly interleaved. Therefore, in order to reuse previous computations, it is necessary not only to keep track of the relevant previous computations, but also to appropriately track the interleaving. The algorithms described in this paper can correctly compute the network output for all these described complex cases, while ensuring that the convolutional operations are performed no more than once even across multiple windows.

[0055] The first algorithm discussed below allows the TCN to be executed in real time while eliminating redundant convolutions.

[0056]

[0057] In Algorithm 1, the number of input channels at each layer is computed in line 1 by combining the number of input channels and the number of all convolutional filters except the last layer. In line 3, the dilated kernel size k′ for layer I is computed s , and then it is used to initialize a single fixed-size buffer b[i] in the "CW" layout (channel width) in line 4. For the TCN, the width "W" in the "CW" layout refers to the time dimension. In line 5, the write head b h [i] is initialized, which identifies the column along "W" where the next input to the buffer should be written. If a padding size of p is desired, then p zero inputs are fed into Algorithm 2 to appropriately fill the buffer (lines 7 - 11). Setting p = T - 1 ensures that an output is generated on the first input, where T is the receptive field size.

[0058]

[0059]

[0060] Algorithm 2 takes a new single multi-channel input x t and passes it through the network while performing individual convolutions as needed. The new input can trigger at most 1 convolution at each layer of the network, if the input buffer for that layer becomes full. If a convolution is triggered in the output layer, then a new network output is generated.

[0061] In line 1, the new input sample x t is placed into the temporary buffer x′ t . In lines 4 and 5, x′ tCopy it into the input buffer for the current layer. If the buffer is full, then at lines 9 and 10, perform dilated convolution with the activation function and place the result in x′ t to be processed as the input to the next layer. The input buffer is emptied by shifting its layer stride to eliminate the oldest value along the time dimension “W” (line 11), and the write head is updated (line 12). Finally, if convolution is performed in the output layer, then at line 14, copy the temporary buffer x′ t to the output buffer. If the buffer is not full at any layer, then no output is returned and the algorithm terminates (lines 17 and 18).

[0062] To train the TCN, a straightforward approach is to sample mini - batches of the entire input sequence of shape “NCW”, where N is the mini - batch size, C is the input channel size, and W is the “width” of the sequence, and then perform standard stochastic gradient descent optimization on those mini - batches. However, this approach has potential drawbacks. In particular, the data within the sequence may be correlated, meaning that the learning process may be negatively affected unless a sufficiently large mini - batch size is provided. However, if the sequence size is large, it may actually be infeasible to provide a large mini - batch size. Additionally, if the lengths of different sequences are different, then all sequences must be padded to the size of the largest sequence in the mini - batch.

[0063] To address these issues, smaller random subsequences can be extracted and used for training. The smallest reduction that can be used is the size of the receptive field of the network so that only one output is generated. Training the TCN using input subsequences of size equal to its training data is called the fixed - window view of the TCN, and training the TCN using the full sequence is called the continuous view.

[0064] A characteristic of the fixed - window view is that, in order to train the TCN in a way consistent with the Figure 1 continuous view, the architecture of the network can be changed. To illustrate how it can be changed, consider the Figure 1 example TCN shown in. If only a fixed - window length equal to the size of the receptive field is fed into the network, then some of the intermediate convolutions no longer contribute to a single output. If all these uncorrelated convolutions are removed, then the resulting reduced network architecture is the Figure 2 architecture shown in. In this reduced network, the stride of layer 2 is now 2 instead of 1 and the dilation rate of layer 3 is 1 instead of 2. Algorithm 3 presents a straightforward method for pruning unused convolutions in order to generate a pruned TCN for the fixed - window view.

[0065] Note that it is not necessary to perform such reduction in the fixed window view of the TCN. The same stride and network dilation rate specified in the sequential view of the TCN can be used to compute a single network output in the fixed window view of the TCN. Experiments have shown how pruning the network in the fixed window view can reduce the computational requirements when not using RT-TCN. Popular deep learning libraries (such as tensorflow) perform pruning optimization automatically during training.

[0066]

[0067]

[0068] In Algorithm 3, since there is only 1 output, the stride of the output layer is trivial, and since no other input is required other than the input being operated on, the dilation can be reduced to 1 by increasing the stride at the lower layers (lines 1-2). At line 3, compute the stride multiplier (s m ) for the lower layers and start iterating over the lower layers. Using this multiplier, compute the stride and dilation of the layer at lines 5-6. To determine if any input at this layer is not needed, compute the greatest common divisor (GCD) of the updated stride and dilation rate. If the GCD is greater than 1, then it means that some input at this layer is not needed, and at lines 9-10, reduce the stride and dilation rate and set the stride rate multiplier for the lower layers to that GCD (line 7). This process is repeated layer by layer up to the input layer. No reduction is performed at the input layer (line 8) to ensure that the fixed window TCN uses the input at the same rate as specified by the TCN.

[0069] Furthermore, consider the setup of training the TCN using a fixed window representation of the input data and then repeatedly execute the trained network in real time such that an output is desired for every r input samples. To compute the output using RT-TCN, it is necessary to compute the network stride s and dilation rate d in the sequential view of the TCN in order to supply those parameters to the RT-TCN.

[0070] In Algorithm 4, it shows how to compute s and d from the reduced fixed window network with stride s fw [] and network dilation rate d fw [].

[0071]

[0072] In Algorithm 4, the dilation rate multiplier m dIt is initialized to 1 on the first line. On the fourth line, we recalculate the stride of a given layer depending on whether we need to compute the output at a rate different from the stride specified by the fixed window view of the TCN. Next, if necessary, the dilation rate is increased based on the current value of the dilation rate multiplier. Finally, on lines 6 and 7, r is updated to account for whether the stride s at that layer results in a reduction in the rate of a portion of r, and then m is recalculated depending on whether additional convolutional outputs are inserted at the current layer d , thus increasing the dilation rate at higher layers. If a portion of r is not affected by the stride at any layer, then it means there is a stride at the output layer, and the stride rate is updated on line 9.

[0073] Embodiments of the present invention provide an efficient algorithm for computing deep convolutions on time series data in real time, where such windows overlap on subsequent predictions. The algorithm achieves efficiency by ensuring that redundant convolution operations are not performed. The algorithm is also flexible enough to handle different network architectures, where there are many convolutional layers, and each convolutional layer can have a different convolutional kernel size and stride. Such an algorithm should enable efficient execution of time series convolutions on many different platforms where limited computing resources are available.

[0074] Many changes and modifications can be made by those of ordinary skill in the art without departing from the spirit and scope of the present invention. Therefore, it must be understood that the embodiments shown are set forth only for purposes of example and should not be regarded as limiting the present invention as defined by the appended claims. For example, although the elements of the claims are set forth below in a certain combination, it must be expressly understood that the present invention includes fewer, more, or different other combinations of the disclosed elements.

[0075] The words used in this specification to describe the present invention and its various embodiments should be understood not only in their commonly defined sense, but also by special definition in this specification to include a general structure, material, or action representing a single species.

[0076] Therefore, the definitions of the words or elements of the following claims are defined in this specification to include not only combinations of elements literally set forth. Thus, in this sense, it is contemplated that any one element in the following claims may be equivalently replaced by two or more elements, or two or more elements in the claims may be replaced by a single element. Although the elements may be described above as acting in certain combinations and even initially claimed as such, it should be expressly understood that in some cases, one or more elements from the claimed combination may be deleted from the combination, and the claimed combination may be indicated as a sub - combination or a variant of a sub - combination.

[0077] Changes which are not material to the claimed subject matter as viewed by a person of ordinary skill in the art are expressly contemplated as being equivalently within the scope of the claims. Accordingly, obvious substitutions now or later known to one of ordinary skill in the art are defined as being within the scope of the defined elements.

[0078] Accordingly, the claims are to be understood to include what is specifically shown and described above, what is conceptually equivalent, what can be obviously substituted, and also what incorporates the essential idea of the invention.

Claims

1. A computerized method for performing deep convolution on time series data in a temporal convolutional network, comprising: Inputting time series data of a first fixed time window; Predicting a first output through a trained deep convolutional network; Inputting time series data of a second fixed time window; Predicting a second output through a trained deep convolutional network, wherein The convolutional operations used to provide the first output and required for the second output are not repeated for the prediction of the second output or for subsequent predicted outputs; The method further comprises retaining in a memory the inputs and intermediate values required for performing future convolutional operations; Wherein a single convolutional operation is performed only once across time series data of multiple fixed time windows; and Wherein the outputs of previous convolutional operations are reused as needed in all layers of the temporal convolutional network to minimize the computational requirements and memory footprint of the temporal convolutional network during its real-time evaluation.

2. The method according to claim 1, further comprising performing a ready convolutional operation once all inputs required for performing the ready convolutional operation are available.

3. The method according to claim 1, further comprising: Training a temporal convolutional network via a fixed window view, wherein the temporal convolutional network is trained using fixed-time concatenations of the input time series; And Performing the temporally convolutional network trained via the fixed window view in real time.

4. A computerized method for performing deep convolution on time series data in a temporal convolutional network, comprising: Inputting time series data of a first fixed time window; Performing a ready convolutional operation once all inputs required for performing the ready convolutional operation are available; Retaining in a memory the inputs and intermediate values required for performing future convolutional operations, wherein a single convolutional operation is performed only once across time series data of multiple fixed time windows; Predicting a first output through a trained deep convolutional network; Inputting time series data of a second fixed time window; and Predicting a second output through a trained deep convolutional network, wherein the convolutional operations required for the second output retained in the memory are reused instead of re-performing the convolutional operations; Wherein the outputs of previous convolutional operations are reused as needed in all layers of the temporal convolutional network to minimize the computational requirements and memory footprint of the temporal convolutional network during its real-time evaluation.

5. The method according to claim 4, further comprising: Training a temporal convolutional network via a fixed window view, wherein the temporal convolutional network is trained using fixed-time concatenations of the input time series; And Performing the temporally convolutional network trained via the fixed window view in real time.

6. A non-transitory computer-readable storage medium having stored thereon an executable program, wherein the program instructs one or more processors to perform actions, including: Inputting time series data of a first fixed time window; Predicting a first output through a trained deep convolutional network; Inputting time series data of a second fixed time window; Predicting a second output through a trained deep convolutional network, wherein The convolutional operations used to provide the first output and required for the second output are not repeated for the prediction of the second output or for subsequent predicted outputs; The operation further includes retaining in a memory inputs and intermediate values required for performing future convolution operations; wherein a single convolution operation is performed only once across time series data of multiple fixed time windows; and wherein outputs of previous convolution operations are reused as needed in all layers of the temporal convolutional network to minimize computational requirements and memory footprint of the temporal convolutional network during its real-time evaluation.

7. The non-transitory computer-readable storage medium storing an executable program according to claim 6, wherein the program instructs one or more processors to perform further operations including performing a ready convolution operation once all inputs required for performing the ready convolution operation are available.

8. The non-transitory computer-readable storage medium storing an executable program according to claim 6, wherein the program instructs one or more processors to perform further operations including: training a temporal convolutional network via a fixed window view, wherein the temporal convolutional network is trained using a fixed-time concatenation of an input time series; and performing in real time the temporal convolutional network trained via the fixed window view.

Images