Method, computer system, computer program (multi-channel time series analysis using a transformer with a multivariate parallel attention model)

Abstract

The system provides a mechanism for performing prediction or classification tasks. [Solution] According to one embodiment of the present invention, a system for performing a prediction task or a classification task comprises one or more memories and at least one processor coupled to the one or more memories. The system generates a plurality of tokens from a plurality of multichannel inputs. A plurality of embeddings are generated from the plurality of tokens by an encoder. The prediction task or the classification task is performed by a multivariate parallel attention model based on the plurality of embeddings. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel. Embodiments of the present invention further include a method and a computer program for performing a prediction task or a classification task in substantially the same manner as described above.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to machine learning, and more specifically, to performing time-series analysis of multi-channel input data using a transformer with a multivariate parallel attention model configured to determine time-based attention, channel-based attention, and content-based attention in parallel. [Overview of the project] [Problems that the invention aims to solve]

[0002] Multichannel input data (e.g., multivariate time series data) can be analyzed to identify patterns and interrelationships, which can then be used to generate predictions and inform decision-making. For example, multichannel input data such as electroencephalogram (EEG) recordings of neuronal activity provide insightful information for monitoring brain states and detecting changes in brain activity. However, conventional transformer-based methods struggle to generalize across heterogeneous data with varying numbers of channels because these models cannot simultaneously process the temporal and spatial dimensions of multivariate time series data at an attentional level. Furthermore, conventional channel-independent methods do not share information across channels, while conventional channel-mixed methods process time series in multiple steps. These methods lack the ability to capture and integrate the temporal and spatial aspects of multivariate time series data at an attentional level and efficiently leverage transformer-based models to generate accurate predictions and / or classifications of multichannel input data. [Means for solving the problem]

[0003] According to one embodiment of the present invention, a system for performing a prediction task or a classification task comprises one or more memories and at least one processor coupled to the one or more memories. The system generates a plurality of tokens from a plurality of multichannel inputs. A plurality of embeddings are generated from the plurality of tokens by an encoder. The prediction task or the classification task is performed by a multivariate parallel attention model based on the plurality of embeddings. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel. Embodiments of the present invention further include a method and a computer program product for performing a prediction task or a classification task in substantially the same manner as described above. [Brief explanation of the drawing]

[0004] Generally, similar reference numerals in various diagrams are used to specify similar components.

[0005] [Figure 1] This is a schematic diagram illustrating an example of a computing environment according to one embodiment of the present invention.

[0006] [Figure 2] This is a flowchart illustrating how a transformer can perform prediction and / or classification tasks according to one embodiment of the present invention.

[0007] [Figure 3] This is a schematic diagram illustrating the tokenization of multi-channel time series according to one embodiment of the present invention.

[0008] [Figure 4] This is a flowchart illustrating how to perform prediction and / or classification tasks using a multivariate parallel attention model according to one embodiment of the present invention.

[0009] [Figure 5A]This is a schematic diagram illustrating how an encoder generates multiple embeddings from multiple tokens, according to one embodiment of the present invention.

[0010] [Figure 5B] This is pseudocode providing an example of a method for generating multiple embeddings from multiple tokens, according to one embodiment of the present invention.

[0011] [Figure 6A] This is a schematic diagram illustrating how a multivariate parallel attention model generates one or more classification and / or prediction outputs from multiple embeddings, according to one embodiment of the present invention.

[0012] [Figure 6B] This is pseudocode providing an example of a method for performing decoding in a multivariate parallel attention model according to one embodiment of the present invention.

[0013] [Figure 6C] This is pseudocode providing an example of a multilayer perceptron scheme in a multivariate parallel attention model according to one embodiment of the present invention.

[0014] [Figure 7A] This is a schematic diagram of a self-attention mechanism for processing time-series data according to one embodiment of the present invention.

[0015] [Figure 7B] This is pseudocode providing an example of a method for implementing a multivariate parallel attention model according to one embodiment of the present invention.

[0016] [Figure 7C] This is a schematic diagram comparing a structured dropout operation according to one embodiment of the present invention with a conventional dropout operation.

[0017] [Figure 7D]This is pseudocode providing an example of a method for implementing a flash multivariate parallel attention model according to one embodiment of the present invention.

[0018] [Figure 8A] This is a schematic diagram illustrating how a prediction task is performed using a multivariate parallel attention model according to one embodiment of the present invention.

[0019] [Figure 8B] This is a schematic diagram illustrating the training of a multivariate parallel attention model using one embodiment of the present invention.

[0020] [Figure 8C] This is a schematic diagram illustrating how a prediction task is performed using a multivariate parallel attention model based on test data, according to one embodiment of the present invention.

[0021] [Figure 8D] This is pseudocode providing an example of a method for performing an inference task using a multivariate parallel attention model, according to one embodiment of the present invention. [Modes for carrying out the invention]

[0022] Multichannel input data (e.g., multivariate time series data) can be analyzed to identify patterns and interrelationships, which can then be used to generate predictions and inform decision-making. For example, multichannel input data such as EEG recordings of neuronal activity can provide insightful information for monitoring brain states and detecting changes in brain activity. However, conventional transformer-based methods struggle to generalize across heterogeneous data with varying numbers of channels because these models cannot simultaneously process the temporal and spatial dimensions of multivariate time series data at an attentional level. Furthermore, conventional channel-independent methods do not share information across channels, while conventional channel-mixed methods process time series in multiple steps. These methods lack the ability to capture and integrate the temporal and spatial aspects of multivariate time series data at an attentional level and efficiently leverage transformer-based models to generate accurate predictions and / or classifications of multichannel input data.

[0023] Therefore, one embodiment of the present invention efficiently and accurately performs a prediction or classification task. Embodiments of the present invention utilize a transformer with a multivariate parallel attention model to determine time-based attention, channel-based attention, and content-based attention in parallel. Multiple tokens are generated from the multiple multichannel inputs, and multiple embeddings are generated from the multiple tokens. Based on the multiple embeddings, the prediction or classification task is performed by the multivariate parallel attention model. Embodiments of the present invention further sum the time-based attention, channel-based attention, and content-based attention to determine an attention value for training the multivariate parallel attention model. This provides a multivariate parallel attention model that can effectively capture contextual information by simultaneously directing attention to the time, channel, and content of the input data, thereby enabling improved performance of classification and / or prediction tasks and efficient use of computational resources.

[0024] Typically, EEG signal data can be collected from electrodes placed on or within the human brain (scalp EEG or iEEG). One embodiment of the present invention utilizes a transformer with a multivariate parallel attention model to generate prediction and / or classification of iEEG signal data (e.g., multi-channel time-series data) that represent brain activity. For example, iEEG signal data may be divided into multiple time windows, each time window being divided into multiple segments such that each segment is associated with a specific time and a specific channel (e.g., electrode). The multiple segments are processed by a feature extraction technique to generate multiple tokens. The multiple tokens are processed by an encoder to generate multiple embeddings representing the multiple tokens. One embodiment of the present invention provides the multiple embeddings to a multivariate parallel attention model to simultaneously predict the next time-series segment representing future neuronal activity and detect seizures. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel.

[0025] One embodiment of the present invention utilizes a transformer with a multivariate parallel attention model to perform a prediction or classification task. Multiple multichannel inputs are divided into multiple time windows, each containing a subset of the multiple multichannel inputs associated with time. The multiple time windows are divided into multiple segments, each associated with channels represented within the multiple multichannel inputs. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel. Time-based attention is determined based on the temporal distance between each of the multiple segments and between other segments of the multiple segments. Channel-based attention is determined based on the spatial distance between each of the multiple segments and between other segments. Content-based attention is determined by applying a self-attention mechanism to the multiple segments. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel.

[0026] According to an aspect of the present invention, a method for performing a prediction task or a classification task is provided. At least one processor generates a plurality of tokens from a plurality of multichannel inputs. The at least one processor generates a plurality of embeddings from the plurality of tokens using an encoder. The at least one processor performs the prediction task or the classification task based on the plurality of embeddings using a multivariate parallel attention model. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel.

[0027] This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multi-channel input data. By leveraging a multivariate parallel attention model configured to determine time-based attention, channel-based attention, and content-based attention in parallel, embodiments of the present invention reduce the computational complexity associated with transformer-based analysis of multivariate time series. This mitigates the problem of difficulty in handling data of varying lengths and dimensions, overcomes computational barriers, and ultimately leads to more efficient use of computational resources and higher computational power. Furthermore, embodiments of the present invention provide a multivariate parallel attention model that can effectively capture contextual information by simultaneously paying attention to the time, channel, and content of input data, thereby enabling improved performance in classification and / or prediction tasks.

[0028] In this embodiment, the plurality of multichannel inputs include multichannel time-series data, and the step of generating the plurality of tokens comprises dividing the plurality of multichannel inputs into a plurality of time windows, each containing a subset of the plurality of multichannel inputs associated with time; dividing each of the plurality of time windows into a plurality of segments, each associated with channels represented in the plurality of multichannel inputs; and processing the plurality of segments by a feature extraction technique to generate the plurality of tokens. This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multichannel input data.

[0029] In an embodiment, the at least one processor further trains the multivariate parallel attention model to perform the prediction task and the classification task simultaneously, the prediction task comprising predicting one or more data points following the plurality of multichannel inputs, and the classification task comprising classifying categories associated with the plurality of multichannel inputs. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multichannel input data.

[0030] In an embodiment, the step of training the multivariate parallel attention model includes a step of summing the time-based attention, the channel-based attention, and the content-based attention to determine the attention value for training the multivariate parallel attention model. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multi-channel input data. Furthermore, embodiments of the present invention provide a multivariate parallel attention model that can effectively capture contextual information by simultaneously directing attention to the time, channel, and content of the input data, thereby enabling improved performance in classification and / or prediction tasks.

[0031] In this embodiment, the multivariate parallel attention model includes a decoder and a multilayer perceptron. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multi-channel input data.

[0032] In the embodiment, time-based attention is determined based on the temporal distance between each of the multiple segments and between other segments of the multiple segments; channel-based attention is determined based on the spatial distance between each of the multiple segments and between other segments; and content-based attention is determined by applying a self-attention mechanism to the multiple segments, with the temporal distances stored in a temporal-spatial codebook and the spatial distances stored in a spatial-spatial codebook. This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multi-channel input data.

[0033] In this embodiment, one of the plurality of tokens is randomly selected as a question token representing the classification objective, and one or more of the plurality of tokens are selected as answer tokens representing the classification output. This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multi-channel input data.

[0034] According to an aspect of the present invention, a computer system is provided for performing a prediction task or a classification task, comprising a processor set, one or more computer-readable storage media, and program instructions stored on the one or more computer-readable storage media, which cause the processor set to perform an operation. The program instructions cause the processor set to generate a plurality of tokens from a plurality of multichannel inputs. The program instructions cause the processor set to generate a plurality of embeddings from the plurality of tokens using an encoder. The program instructions cause the processor set to perform the prediction task or the classification task based on the plurality of embeddings using a multivariate parallel attention model. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel.

[0035] This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multi-channel input data. By leveraging a multivariate parallel attention model configured to determine time-based attention, channel-based attention, and content-based attention in parallel, embodiments of the present invention reduce the computational complexity associated with transformer-based analysis of multivariate time series. This mitigates the problem of difficulty in handling data of varying lengths and dimensions, overcomes computational barriers, and ultimately leads to more efficient use of computational resources and higher computational power. Furthermore, embodiments of the present invention provide a multivariate parallel attention model that can effectively capture contextual information by simultaneously paying attention to the time, channel, and content of input data, thereby enabling improved performance in classification and / or prediction tasks.

[0036] In the embodiment of the computer system, the plurality of multichannel inputs include multichannel time-series data, and the procedure for generating the plurality of tokens comprises: dividing the plurality of multichannel inputs into a plurality of time windows, each containing a subset of the plurality of multichannel inputs associated with time; dividing each of the plurality of time windows into a plurality of segments, each associated with channels represented in the plurality of multichannel inputs; and processing the plurality of segments by a feature extraction technique to generate the plurality of tokens. This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multichannel input data.

[0037] In the embodiment of the computer system, the program instructions further train the processor set to perform the multivariate parallel attention model to execute the prediction task and the classification task simultaneously, wherein the prediction task includes predicting one or more data points following the plurality of multichannel inputs, and the classification task includes classifying categories associated with the plurality of multichannel inputs. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multichannel input data.

[0038] In the embodiment of the computer system described above, the procedure for training the multivariate parallel attention model includes a step of summing the time-based attention, the channel-based attention, and the content-based attention to determine the attention value for training the multivariate parallel attention model. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multi-channel input data. Furthermore, embodiments of the present invention provide a multivariate parallel attention model that can effectively capture contextual information by simultaneously directing attention to the time, channel, and content of the input data, thereby enabling improved performance in classification and / or prediction tasks.

[0039] In the embodiment of the computer system described above, the multivariate parallel attention model includes a decoder and a multilayer perceptron. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multichannel input data.

[0040] In the embodiment of the computer system, time-based attention is determined based on the temporal distance between each of the multiple segments and between other segments of the multiple segments; channel-based attention is determined based on the spatial distance between each of the multiple segments and between other segments; and content-based attention is determined by applying a self-attention mechanism to the multiple segments, with the temporal distances stored in a temporal-spatial codebook and the spatial distances stored in a spatial-spatial codebook. This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multi-channel input data.

[0041] In the embodiment of the computer system described above, one of the plurality of tokens is randomly selected as a question token representing the classification objective, and one or more of the plurality of tokens are selected as answer tokens representing the classification output. This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multi-channel input data.

[0042] According to an aspect of the present invention, a computer program product for performing a prediction task or a classification task is provided. The computer program product comprises one or more computer-readable storage media, and program instructions stored on the one or more computer-readable storage media for performing an operation. The operation includes a procedure for generating a plurality of tokens from a plurality of multichannel inputs. The operation includes a procedure for generating a plurality of embeddings from the plurality of tokens using an encoder. The operation includes a procedure for performing the prediction task or the classification task based on the plurality of embeddings using a multivariate parallel attention model. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel.

[0043] This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multi-channel input data. By leveraging a multivariate parallel attention model configured to determine time-based attention, channel-based attention, and content-based attention in parallel, embodiments of the present invention reduce the computational complexity associated with transformer-based analysis of multivariate time series. This mitigates the problem of difficulty in handling data of varying lengths and dimensions, overcomes computational barriers, and ultimately leads to more efficient use of computational resources and higher computational power. Furthermore, embodiments of the present invention provide a multivariate parallel attention model that can effectively capture contextual information by simultaneously paying attention to the time, channel, and content of input data, thereby enabling improved performance in classification and / or prediction tasks.

[0044] In the embodiment of the computer program product, the plurality of multichannel inputs include multichannel time-series data, and the procedure for generating the plurality of tokens comprises: dividing the plurality of multichannel inputs into a plurality of time windows, each containing a subset of the plurality of multichannel inputs associated with time; dividing each of the plurality of time windows into a plurality of segments, each associated with channels represented in the plurality of multichannel inputs; and processing the plurality of segments by a feature extraction technique to generate the plurality of tokens. This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multichannel input data.

[0045] In an embodiment of the computer program product, the operation further comprises a step of training the multivariate parallel attention model to perform the prediction task and the classification task simultaneously, wherein the prediction task includes predicting one or more data points following the plurality of multichannel inputs, and the classification task includes classifying categories associated with the plurality of multichannel inputs. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multichannel input data.

[0046] In the embodiment of the computer program product, the procedure for training the multivariate parallel attention model includes a procedure for determining an attention value for training the multivariate parallel attention model by summing the time-based attention, the channel-based attention, and the content-based attention. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multi-channel input data. Furthermore, embodiments of the present invention provide a multivariate parallel attention model that can effectively capture contextual information by simultaneously directing attention to the time, channel, and content of the input data, thereby enabling improved performance in classification and / or prediction tasks.

[0047] In the embodiment of the computer program product described above, the multivariate parallel attention model includes a decoder and a multilayer perceptron. This results in an enhanced attention model capable of efficiently and accurately generating predictions and classifications of multichannel input data.

[0048] In the embodiment of the computer program product, time-based attention is determined based on the temporal distance between each of the multiple segments and between other segments of the multiple segments; channel-based attention is determined based on the spatial distance between each of the multiple segments and between other segments; content-based attention is determined by applying a self-attention mechanism to the multiple segments; the temporal distance is stored in a temporal-spatial codebook; and the spatial distance is stored in a spatial-spatial codebook. This results in an enhanced attention model capable of efficiently and accurately generating predictions and / or classifications of multi-channel input data.

[0049] In an exemplary scenario, a multivariate parallel attention model can be used to generate predictions and / or classifications of iEEG signal data (e.g., multi-channel time-series data) representing brain activity. For example, iEEG signal data may be divided into multiple time windows, each time window being divided into multiple segments such that each segment is associated with a specific time and a specific channel (e.g., an electrode). The multiple segments are processed by a feature extraction technique to generate multiple tokens. The multiple tokens are processed by an encoder to generate multiple embeddings representing the multiple tokens. One embodiment of the present invention provides the multiple embeddings to a multivariate parallel attention model to simultaneously predict the next time-series segment representing future neuronal activity and detect sequences. Another embodiment of the present invention provides a multivariate parallel attention model configured to determine time-based attention, channel-based attention, and content-based attention in parallel, thereby reducing computational complexity and effectively capturing contextual information to improve performance on classification and prediction tasks.

[0050] Various aspects of this disclosure are described by explanatory text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of computer program products (CPPs). With respect to any flowchart, operations may be performed in a different order than those shown in a given flowchart, depending on the technology involved. For example, again, depending on the technology involved, two operations shown in consecutive blocks of a flowchart may be performed in reverse order, as a single integrated stage, simultaneously, or with at least partial time overlap.

[0051] Embodiments of a computer program product ("CPP Embodiment" or "CPP") are terms used in this disclosure to describe any set of one or more storage media (also called "mediums") that are collectively contained in a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may be, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any preferred combination thereof. Some known types of storage devices, including these media, include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as pits / lands formed on the main surface of a punch card or disk), or any suitable combination of the foregoing. When the term "computer-readable storage medium" is used in this disclosure, it shall not be interpreted as storage in the form of a temporary signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmission media.As those skilled in the art will understand, data is typically moved at several intermittent points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection; however, data is not transient while it is stored, and therefore the above does not mean that the storage device is transient.

[0052] Referring to Figure 1, the computing environment 100 includes an example of an environment for executing at least a portion of the computer code involved in carrying out the method of the invention, for example, multi-channel analysis code 200. In addition to block 200, the computing environment 100 includes, for example, a computer 101, a wide area network (WAN) 102, an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor set 110 (including a processing circuit configuration 120 and a cache 121), a communication fabric 111, volatile memory 112, persistent storage 113 (including an operating system 122 and the block 200 identified above), a peripheral device set 114 (including a user interface (UI), a device set 123, storage 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine set 142, a virtual machine set 143, and a container set 144.

[0053] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that is capable of running programs, accessing networks, or querying databases such as remote database 130. As is well understood in the field of computer technology, and depending on the technology, the execution of a computer implementation may be distributed among multiple computers and / or multiple locations. On the other hand, in this description of the computing environment 100, in order to keep the description as concise as possible, the detailed considerations focus on a single computer, specifically computer 101. Computer 101 is not shown in the cloud in Figure 1, but it may be located in the cloud. On the other hand, computer 101 is not required to be located in the cloud, except to any extent that can be definitively shown.

[0054] The processor set 110 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuit configuration 120 may be distributed across multiple packages, for example, multiple interconnected integrated circuit chips. The processing circuit configuration 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is memory located within the processor chip package and is typically used for data or code that should be available for high-speed access by threads or cores running on the processor set 110. The cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuit configuration. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may operate using qubits and be designed to perform quantum computing.

[0055] Computer-readable program instructions are typically loaded onto computer 101, causing the processor set 110 of computer 101 to execute a series of operational steps, thereby enabling the computer implementation method. As a result, the instructions thus executed instantiate the method specified in the flowcharts and / or descriptions of the computer implementation method contained herein (collectively referred to as the "Method of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media considered below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the Method of the Invention. In the computing environment 100, at least some of the instructions for executing the Method of the Invention may be stored in blocks 200 in persistent storage 113.

[0056] The communication fabric 111 is a signal conduction path that enables various components of the computer 101 to communicate with one another. Typically, this fabric consists of switches and conductive paths, such as buses, bridges, physical input / output ports, and similar components. Other types of signal communication paths, such as optical fiber communication paths and / or wireless communication paths, may be used.

[0057] Volatile memory 112 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, but this is not mandatory unless explicitly stated. In computer 101, volatile memory 112 is located in a single package and resides inside computer 101, but alternatively or additionally, volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0058] The persistent storage 113 is any form of non-volatile storage for a computer, currently known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is supplied to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 can take multiple forms, such as various known proprietary operating systems or open-source portable operating system interface type operating systems employing a kernel. The code contained in block 200 typically includes at least a portion of computer code involved in performing the method of the present invention.

[0059] The peripheral device set 114 includes a set of peripheral devices of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 can be implemented in various ways, including Bluetooth® connections, Near-Field Communication (NFC) connections, connections made by cables (such as universal serial bus (USB) type cables), insert-type connections (e.g., secure digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic devices. Storage 124 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 124 may be persistent and / or volatile. In some embodiments, storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 is required to have a large amount of storage (for example, when computer 101 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that may be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0060] The network module 115 is a collection of computer software, hardware, and firmware that enables computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware such as a modem or Wi-Fi® signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for transmitting data over the internet. In some embodiments, the network control function and network forwarding function of the network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control function and forwarding function of the network module 115 are performed on physically separate devices so that the control function manages multiple different network hardware devices. Typically, computer-readable program instructions for carrying out the method of the invention can be downloaded from an external computer or external storage device to computer 101 via a network adapter card or network interface contained within the network module 115.

[0061] WAN102 is any wide area network (e.g., the Internet) that can communicate computer data over non-local distances using any currently known or future-developed technology for communicating computer data. In some embodiments, WAN102 may be replaced and / or complemented by a local area network (LAN), such as a Wi-Fi® network, designed to communicate data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0062] The end-user device (EUD) 103 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 101), and may take any of the forms considered above in relation to computer 101. The EUD 103 typically receives useful and valuable data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to the end-user, these recommendations would typically be transmitted from computer 101's network module 115 to the EUD 103 via the WAN 102. Thus, the EUD 103 can display or otherwise present recommendations to the end-user. In some embodiments, the EUD 103 may be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0063] The remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 101. For example, in a hypothetical case where computer 101 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 101 from the remote database 130 of the remote server 104.

[0064] The public cloud 105 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without requiring direct and active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that make up the host physical machine set 142, which is a universe of physical computers located within and / or available to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs may be stored as images and are understood to be transferable either as images or after VCE instantiation between hosts on various physical machines. The cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instantiations, and manages the active instantiation of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate over the WAN 102.

[0065] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature in the kernel that allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave as actual computers from the perspective of the programs running in them. Computer programs running on a normal operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to that container; this feature is known as containerization.

[0066] The private cloud 106 is similar to the public cloud 105, except that its computing resources are available only for use by a single enterprise. While the private cloud 106 is shown as being in communication with the WAN 102, in other embodiments, the private cloud may be completely isolated from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a separate discrete entity, but the larger hybrid cloud architecture is coupled together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configuration clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0067] Cloud computing services and / or microservices (not shown separately in Figure 1): Public and private clouds 105, 106 are programmed and configured to deliver cloud computing services and / or microservices (unless otherwise stated, the term “microservices” is interpreted to encompass larger “services,” regardless of size). Cloud services are typically infrastructure, platforms, or software hosted by a third-party provider and made available to users over the internet. Cloud services facilitate the flow of user data from front-end clients (e.g., user-side servers, tablets, desktops, laptops) to the provider’s systems and back over the internet. In some embodiments, cloud services may be configured and orchestrated according to an “as-a-service” technology paradigm, which is presented to internal or external customers in the form of cloud computing services. An as-a-service offering typically provides endpoints that various customers interface with. These endpoints are typically based on a set of APIs. One category of as-a-service offerings is Platform as a Service (PaaS), where the service provider provisions, instantiates, runs, and manages modular bundles of code that customers can use to instantiate a computing platform and one or more applications without the complexity of building and maintaining the infrastructure typically associated with them. Another category is Software as a Service (SaaS), where software is centrally hosted and allocated on a subscription basis. SaaS is also known as on-demand software, web-based software, or web-hosted software. Four technical sub-areas involved in cloud services are: deployment, integration, on-demand, and virtual private networks.

[0068] A flowchart relating to a transformer 202 for performing prediction and / or classification tasks (e.g., by a multichannel analysis code 200, computer 101, etc.) according to one embodiment of the present invention is illustrated in Figure 2. First, multichannel input data 205 is acquired. The multichannel input data 205 may include multichannel time series data. Multichannel time series data includes multiple time series, each having one or more dimensions. For example, multichannel time series data may be multivariate, where one dimension of the data represents time, and one or more second dimensions represent time-dependent variables. For example, biosignals such as electrical activity of the brain captured by an EEG or electrical activity of the heart captured by an electrocardiogram (ECG) may be represented as a multichannel time series having a first dimension representing time and one or more second dimensions representing channels (e.g., electrodes). In another example, temperature and humidity data over time in a city may be represented as a multichannel time series in which temperature and humidity are channels having data that changes over time. Multichannel time series data can be fixed-length or variable-length. For example, variable-length multichannel time series data may contain a set of time series, each having a different number of time points.

[0069] The multi-channel input data 205 is provided as input to the tokenizer 210 of the transformer 202. The tokenizer 210 may perform a two-dimensional (2D) tokenization procedure, which converts the multi-channel input data 205 into multiple one-dimensional (1D) vectors representing multiple tokens 220. For example, multi-channel time series data may be partitioned by the 2D tokenization procedure to generate multiple segments of the time series data representing multiple tokens 220. The multiple tokens 220 are provided as input to the encoder 225 of the transformer 202 to generate one or more embeddings 230. The one or more embeddings 230 are provided to the multivariate parallel attention model 235 of the transformer 202 to generate one or more predictive and / or classification outputs 240. One or more predictive and / or classification outputs 240 are compared to the actual output (e.g., ground truth value) in the loss calculation operation 245 to determine the loss 250. The loss 250 can be iteratively optimized to train the multivariate parallel attention model 235 until a stopping criterion is met. The stopping criterion may be based on model performance, the number of iterations, or any preferred criterion defined and / or configured by the user.

[0070] A tokenization procedure 300, according to one embodiment of the present invention, which is used to generate multiple tokens from a multichannel time series 305 (for example, by a multichannel analysis code 200, a computer 101, etc.), is illustrated in Figure 3. The multichannel time series 305 may be represented by a 2D matrix of independent segments that can be transformed into multiple 1D vectors. First, the multichannel time series 305 is partitioned into multiple time windows. Each time window contains a subset of data in the multichannel time series 305 corresponding to all channels associated with a particular time. For example, a time window 310 extracted from the multichannel time series 305 contains data from channel 1 ("CH1"), channel 2 ("CH2"), and channel 3 ("CH3") at time T1. Furthermore, the multichannel time series 305 may be partitioned into multiple per-channel segments. Each per-channel segment contains a subset of data in the multichannel time series 305 corresponding to one specific channel at all time points. For example, the channel-specific segments 315 extracted from the multi-channel time series 305 contain data from CH3 at all time points (e.g., T1, T2, T3, T4, etc.).

[0071] Each time window (e.g., time window 310) is partitioned to generate multiple segments. Each of these segments corresponds to data in a multichannel time series 305 associated with a specific time and a specific channel. For example, time window 310 is partitioned channel by channel into multiple segments, such as segment 320 containing time series data associated with CH3 at T1. As an example, a multichannel time series 305 may be partitioned into multiple segments, T1-CH1, T1-CH2, T1-CH3, T2-CH1, ..., T4-CH3. Each segment is considered a token that can be processed by a model (e.g., a large-scale language model). The tokenization procedure is applied channel by channel, meaning that each segment remains one-dimensional. Each channel is processed independently and in parallel by the tokenization procedure, thus significantly reducing computational complexity. In a particular embodiment, one or more segments may be extracted from multiple channel-by-channel segments (e.g., channel-by-channel segments 315).

[0072] A method 400 for performing prediction and / or classification tasks by a transformer with a multivariate parallel attention model (e.g., by a multichannel analysis code 200, computer 101, etc.), according to one embodiment of the present invention, is illustrated in Figure 4. First, multichannel time series data 405 is acquired. For example, the multichannel time series data 405 may include a set of time series, each corresponding to a specific channel. The multichannel time series data 405 may be tokenized by a tokenization procedure (e.g., tokenization procedure 300) to generate a plurality of tokens (e.g., 1D vectors) representing multiple segments of the multichannel time series data 405. For each time series in the multichannel time series data 405, a token (e.g., 1D vector) may be randomly selected as a question token (or "Q token") and attached to a token (e.g., signal data) representing the time series. The question token represents a classification objective associated with the classification task. For example, a seizure detection classification task may be performed on multi-channel time-series data 405 (e.g., intracranial electroencephalography (iEEG) signals) for the classification purpose of detecting the number of seizure occurrences. Furthermore, multiple 1D vectors are selected to represent multiple answer tokens representing possible classification outputs. The multiple tokens representing the multi-channel time-series data 405, along with multiple question tokens, form multiple tokens 410 that are provided as input to the transformer's encoder 415. The output corresponding to each token representing a data point in the time series is the next token in that time series. The output corresponding to a question token is an answer token indicating a classification output (e.g., absence or presence of seizure).

[0073] In embedding operation 420, encoder 415 generates multiple embeddings 425 based on multiple tokens 410. Multiple tokens 410 are provided as input to encoder 415, which is configured to apply a feature extraction procedure to generate multiple feature vectors. The feature extraction procedure can be implemented by any prior art and / or model, including wavelet synthesis, convolutional neural networks, multilayer perceptrons, etc. In a particular embodiment, each of the multiple feature vectors is projected onto a lower-dimensional subspace to ensure efficient processing. For example, each segment (represented by tokens) extracted from a time window independently undergoes wavelet decomposition, which is then linearly projected onto a smaller space depending on the size of the overall model. This projection, or feature vector, results in embeddings corresponding to each of the multiple tokens 410, and thus the multiple embeddings 425 are formed.

[0074] Multiple embeddings 425 are provided as input to a multivariate parallel attention model 430 of the transformer, which is configured to perform classification and / or prediction tasks. The multivariate parallel attention model 430 may be implemented by a multivariate parallel attention model 432 configured to determine time-based attention, channel-based attention, and content-based attention in parallel. Multiple outputs 435 are generated by the multivariate parallel attention model 430. The output corresponding to each token representing a data point in a time series is the next token in that time series. The output corresponding to a question token is an answer token indicating a classification output (e.g., absence or presence of a seja). In addition to performing classification tasks, the multivariate parallel attention model 430 may perform prediction tasks, such as predicting time-series segments representing future neuronal activity (e.g., iEEG signals). In certain embodiments, the multivariate parallel attention model 430 may perform classification and prediction tasks simultaneously.

[0075] Multiple outputs 435 are compared in the loss calculation 440 with multiple target outputs ("targets") and multiple confounding outputs ("confounding factors"). Target outputs refer to the actual outputs that take the dataset into account. For example, in a prediction task performed by a multivariate parallel attention model 430 to predict future time series segments, the predicted output O of the multiple outputs 435 is... 1,1 (For example, the output associated with channel 1 at time 1) is the target embedding E 1,2 This can be compared to target 450, such as (for example, the embedding associated with channel 1 at time 2). Expected output O Q (For example, an output associated with a question token) can be compared to target 460. Multiple confounding factors can be selected for each of the above multiple outputs 435. For example, multiple confounding factors 445 can be used to predict output O 1,1 Corresponding to this, multiple confounding factors 455 predict the output O Q It corresponds to.

[0076] Multiple time windows may be selected from multi-channel time series data 405 to form a batch of time windows. Multiple input segments are randomly sampled from the batch of time windows to generate multiple confounding factors, including multiple confounding factors 445 and multiple confounding factors 455. The sampled input segments (e.g., confounding factors 445 and 455) represent actual data from multi-channel time series data 405 (e.g., actual iEEG signals) that are expected to be very different from the true target in order to balance between too high and too low similarity between the confounding factors and the multiple outputs 435.

[0077] Multiple confounding factors 445 and 455 are provided for loss calculation 440. The predicted output (e.g., multiple outputs 435) is compared with the actual output (e.g., targets 450 and 460) and the confounding factors (e.g., multiple confounding factors 445 and 455) to determine the control loss. The control loss is configured to increase the cosine similarity of the predicted output to the true target while decreasing the cosine similarity of the predicted output to the confounding target. Based on the control loss, the multivariate parallel attention model 430 is iteratively optimized as training progresses to produce predicted outputs similar to the encoded segment (e.g., input to the multivariate parallel attention model 430). The multivariate parallel attention model 430 becomes increasingly capable of selecting the correct target, and consequently, predicting future tokens (e.g., signals).

[0078] An encoder 500A, configured to generate multiple embeddings from multiple tokens (e.g., by a multi-channel analysis code 200, computer 101, etc.) according to one embodiment of the present invention, is illustrated in Figure 5A. Multiple tokens 505 representing multiple segments of multi-channel input data (e.g., multi-channel time-series data) are provided as input to encoder 500A, which includes a first layer configured to perform a Daubechies 4 wavelet (db4) wavelet operation 510, a second layer configured to perform a root mean square layer normalization (RMSNorm) operation 515, and a third layer for performing a linear operation 520 (e.g., a linear layer). A subset of the multiple tokens 505 representing a particular segment in the multi-channel time-series data passes through encoder 500A independently. First, each subset of the above-mentioned tokens 505 is processed by a db4 wavelet operation 510 (e.g., db4 wavelet decomposition) to dynamically preserve both high and low frequencies of multi-channel time-series data (e.g., iEEG signal) with varying resolutions. For example, wavelet decomposition may be used to preserve high-frequency oscillations that represent important aspects of the iEEG signal.

[0079] The output from the db4 wavelet operation 510 is processed by the RMSNorm operation 515, which includes normalizing the activations by dividing them by their root mean square values. The output from the RMSNorm operation 515 is then linearly projected into a lower-dimensional space in the linear operation 520, resulting in multiple feature vectors representing multiple embeddings 525. The processing (e.g., operations 510, 515, and 520) is repeated for each segment within a particular time window, thereby forming multiple embeddings 525 that can be provided as input to a transformer-based model. The transformer-based model can leverage any transformer architecture configured to generate classification and / or prediction of input data. A transformer-based architecture (e.g., the LLama2 architecture) can provide a generative model that is powerful enough to process brain iEEG signals and computationally light enough to allow for large-scale testing. For example, a transformer-based architecture may contain millions of parameters (e.g., 75 million parameters). Multiple embeddings 525 can be learnable embeddings, or in other words, they can transform high-dimensional data into low-dimensional data while preserving important characteristics of the data.

[0080] Figure 5B illustrates a pseudocode 500B that provides an exemplary scheme for generating multiple embeddings from multiple tokens (e.g., by a multi-channel analysis code 200, computer 101, etc.) according to one embodiment of the present invention. An exemplary algorithm of pseudocode 500B for implementing the encoder may take multiple raw input segments of a time series (e.g., a multi-channel time series) as input. The number of raw input segments (e.g., n) input Each segment x (=2560) i,j This can be associated with index i, which represents channel C, and index j, which represents time C. Based on multiple unprocessed input segments and a predetermined number of maximum decomposition levels l, the algorithm of pseudocode 500B generates multiple output tokens, and each output token o i,jcan be represented by each of a plurality of embeddings. For example, the plurality of embeddings, where n embed = 768. First, the number of each segment x i,j and the maximum decomposition level l are provided as inputs to the db4 discrete wavelet decomposition operation, resulting in a decomposition output d i,j . Next, the decomposition output d i,j is provided as an input to the RMSNorm operation, which generates a normalization output z i,j . The normalization output z i,j is provided as an input to a linear operation, which generates an output token o i,j . The output token o i,j is the result returned by the algorithm of pseudo-code 500B.

[0081] A system 600A for generating one or more classification and / or prediction outputs from a plurality of embeddings by a transformer with a multivariate parallel attention model (e.g., by multi-channel analysis code 200, computer 101, etc.) according to an embodiment of the present invention is illustrated in FIG. 6A. System 600A includes a multivariate parallel attention model 605 of the transformer. First, a plurality of embeddings 610 (e.g., embeddings generated by encoder 500A) are provided as inputs to the multivariate parallel attention model 605 of the transformer. The multivariate parallel attention model 605 includes a layer configured to perform an RMSNorm operation 615 and a decoder 620 (e.g., a transformer). The plurality of embeddings 610 are processed by the RMSNorm operation 615, which includes normalizing the activations by dividing the activations by their root mean square values. The output from the RMSNorm operation 615 is provided to the decoder 620, which includes a multivariate parallel attention head 625, a layer configured to perform a linear operation 630, and a layer configured to perform a dropout operation 635.

[0082] The multivariate parallel attention head 625 is configured to determine time-based attention, channel-based attention, and content-based attention in parallel. Time-based attention, channel-based attention, and content-based attention can be combined to generate attention values ​​in the self-attention mechanism, thereby enabling the multivariate parallel attention model 605 to focus on various contextual relationships in the input. The multivariate parallel attention model 605 may be a transformer-based model that can effectively capture contextual information, thereby enabling the highest level of performance in many tasks (e.g., classification tasks and / or prediction tasks). For example, the multivariate parallel attention head 625 helps to decompose the structure of the input (e.g., multiple embeddings 610) and better understand the input.

[0083] The output from the multivariate parallel attention head 625 is processed by linear operations 630 and dropout operations 635. The dropout operations 635 may be a technique to improve the generalization ability of neural networks and transformers. For example, dropout is often applied within attention blocks (e.g., decoder 620) to randomly zero out some query-key attentions to avoid the model becoming overly dependent on specific connections. The dropout operations 635 may also be a structured dropout technique configured to blank out entire channels and time points instead of individual segments.

[0084] The multivariate parallel attention model 605 further includes a multi-layer perceptron (MLP) 640. Multiple embeddings 610 are processed by the RMSNorm operation 615 and then provided in parallel to the decoder 620 and the multi-layer perceptron 640, resulting in faster processing compared to sequential processing through the decoder 620 and the multi-layer perceptron 640. The multi-layer perceptron 640 includes a layer configured to perform a first linear operation 645, a sigmoid linear unit (SiLU) 650, a layer configured to perform a second linear operation 655, and a layer configured to perform a dropout operation 660. The output from the RMSNorm operation 615 is processed by the first linear operation 645 (e.g., a linear transformation that maps input data to output data). Next, the output of the first linear operation 645 is processed by SiLU650, which is an activation function computed by a sigmoid function multiplied by its input. Then, another linear operation (e.g., a second linear operation 655) is performed on the output from SiLU650, and the resulting output is processed by a dropout operation 660, which may be a structured dropout technique. The outputs from the RMSNorm operation 615, the decoder 620, and the multilayer perceptron 640 are combined to form the output 665 for the classification and / or prediction task.

[0085] Figure 6B illustrates pseudocode 600B, which provides an exemplary method for implementing a decoder according to one embodiment of the present invention (for example, by a multichannel analysis code 200, a computer 101, etc.). The algorithm of pseudocode 600B for implementing the decoder may take multiple output tokens as input, and each output token o i,j This can be represented by each of several embeddings. For example, the multiple embeddings could be n embed =768. The algorithm in pseudocode 600B involves the RMSNorm operation, which outputs token o i,j It takes as input and normalizes output z i,jThis generates z. Next, the algorithm in pseudocode 600B is configured to compute the attention value using a multi-variate parallel attention (MVPA) mechanism. Normalized decoder output z i,j This is provided as input to a multivariate parallel attention operation, and this is the attention value a i,j Calculate the attention value a. i,j This is provided as input to unbiased linear and dropout operations, with attention output d i,j The following is generated. The feedforward residual may be calculated in parallel with the attention. For example, the normalized decoder output z i,j The feedforward residuals are provided as input to a multilayer perceptron. i,j The following may be generated. Next, the output token o i,j , warning output d i,j , and feedforward residuals s i,j These are summed up, and the decoded output token o i,j The decoded output token o is generated. i,j This is the result returned by the algorithm in pseudocode 600B.

[0086] Figure 6C illustrates pseudocode 600C, which provides an exemplary method for implementing a multilayer perceptron according to one embodiment of the present invention (for example, by a multi-channel analysis code 200, a computer 101, etc.). The algorithm of pseudocode 600C for implementing a multilayer perceptron may take multiple normalized decoder outputs as input, and each normalized decoder output z i,j It is associated with each of the multiple embeddings (for example, n embed =768). Normalized decoder output z i,j Based on this, the pseudocode 600C algorithm outputs the feedforward residual s i,j It is configured to generate the normalized decoder output z. i,j This is processed by an unbiased linear operation, and the output u i,j This is generated. Output u i,j(For example, the normalized decoder output z i,j The output (processed with unbiased linear operation) is then processed with a sigmoid linear unit, resulting in the output g i,j This is generated. Output u i,j and output g i,j This is the inner product of multiple n inner Associated with =1728. Output u i,j and output g i,j The sum is processed by an unbiased linear operation, and the output is a feedforward residual s i,j This is generated. Feedforward residuals s i,j This is the result returned by the algorithm of pseudocode 600C.

[0087] A self-attention mechanism 700A for processing time series data (e.g., by a multi-channel analysis code 200, computer 101, etc.) according to one embodiment of the present invention is illustrated in Figure 7A. A multivariate parallel attention model of the transformer may leverage the self-attention mechanism 700A (e.g., a multivariate parallel attention mechanism) to generate one or more classification and / or prediction outputs based on the input time series. First, time series data 705 (e.g., a multi-channel time series) is processed to select a first segment of the time series that functions as a key 710 and a second segment of the time series that functions as a query 720. The query 720, represented by the self-attention head, may be divided into three components, including content-based attention 730, time-based attention 735, and channel-based attention 740. Content-based attention 730 is configured to determine attention to focus on one or more characteristics of the content (e.g., frequency of the time series data 705). Time-based attention 735 is configured to determine attention based on the time component of the time series data 705. For example, time-based attention 735 may be determined based on the temporal distance between each segment in time-series data 705 and between other segments in time-series data 705. Channel-based attention 740 is configured to determine attention based on channels, space, or components of time-series data 705. For example, channel-based attention 740 may be determined based on the spatial distance between each segment in time-series data 705 and between other segments in time-series data 705. Channel-based attention 740 may also be determined by applying a self-attention mechanism to multiple segments without location encoding. Content-based attention 730, time-based attention 735, and channel-based attention 740 may be summed up to generate an attention value 750 for the self-attention mechanism 700A.

[0088] The self-attention mechanism 700A can be implemented using a single multi-purpose head configured to determine multiple attentions (e.g., time, channel, content, etc.) in parallel, instead of using multiple heads, each focusing on one specific type of attention. Thus, transformer-based models leveraging the self-attention mechanism 700A (e.g., the multivariate parallel attention model 430) can achieve sub-quadratic complexity and improved memory efficiency compared to prior art (e.g., the use of conventional attention mechanisms). In an exemplary embodiment, a transformer-based model leveraging the self-attention mechanism 700A (e.g., the multivariate parallel attention model 430) can achieve a 3.5-fold speedup in processing time compared to prior art.

[0089] Content-based attention 730, time-based attention 735, and channel-based attention 740 each direct attention to different aspects of a signal represented within time-series data 705 (e.g., iEEG signal data). Time-series data 705 can be represented as a 2D set of segments, each associated with a different location in time and space (e.g., a channel). The 2D structure of the set of segments can be maintained by identifying each segment with two distinct indices, c for space and t for time. This representation introduces a priori knowledge of the signal structure into the self-attention mechanism 700A, enabling seamless processing of input data with different numbers of channels without confusing the model (e.g., a multivariate parallel attention model).

[0090] An exemplary conventional attention mechanism is given by equation 1:A i,j =( X i +S_i) T W q T W k T (X j It can be expressed as +S_i), where A i,j represents the attention values ​​at indices i and j, X represents the input (e.g., embedding of time series data 705), and W q T represents the query matrix, Wk T represents the key matrix, S represents the codebook, and T in the upper corner of the matrix indicates the mathematical operation of transposing the matrix. Traditional attention mechanisms have been employed in large-scale language models (LLMs) and have shown unparalleled success in understanding the fundamental properties of natural language. Single-channel data can be treated as equivalent to a sentence by splitting the signal into 1D patches that form tokens. This modality has frequently attracted considerable interest in speech recognition tasks related to the natural language domain. However, there are several drawbacks to applying traditional attention mechanisms to multidimensional inputs (e.g., image data or multi-channel time-series data). For example, flattening patches of multidimensional input into a 1D sequence results in a loss of spatial structure because spatially neighboring patches are not necessarily close in the sequence. Therefore, any information about the structure of the patches is lost. If the image size, the number of patches, and the direction of flattening are kept constant, traditional transformer-based models using traditional attention mechanisms can learn it autonomously. Transformer-based models, once they learn structure, cannot be exposed to different images because they would completely misinterpret them. If a transformer-based model does not learn structure, it lacks crucial information. This results in a less flexible model that cannot easily generalize to different inputs. Another drawback is that traditional transformer-based models do not distinguish between the two dimensions of height and width (for example, they do not distinguish between top, bottom, left, and right). For this reason, traditional transformer-based models that rely on conventional attention mechanisms are unsuitable for classifying and / or predicting multivariate time series data because the two dimensions of time and channel require careful handling.

[0091] EEG signals are multivariate recordings of the brain. Due to the complexity of the data, there are few transformer-based approaches to EEG. In iEEG recordings, subjects may have electrodes directly implanted in multiple areas of the brain for clinical diagnostic purposes. Regarding intracranial implants, neither the location nor even the number of electrodes is standardized. As a result, iEEG is an extremely heterogeneous data modality that is difficult to handle with conventional attentional methods. Channels present a basic source of information because the electric field spreads in different areas of the brain at different time scales and in different intensities, depending on the strength of the connections between areas. Furthermore, relationships between brain regions are not necessarily proportional to their spatial proximity, as distant areas may be more strongly connected than nearby ones. There are extremely complex interactions between space and time that can be learned by the self-attention mechanism 700A, which is configured to handle any possible electrode configuration and clinical setup and to extract as much information as possible from all aspects of the data.

[0092] While conventional attention mechanisms utilize a single codebook, the self-attention mechanism 700A uses a first position codebook that represents space.

number

number

[0093] For example, the self-attention mechanism 700A is given by equation 2:A i,j =( X a_i,b_i +T a_i +C b_i ) T W q T W k T (X a_j,b_j +T a_j +C b_j ) Attention value A can be determined using, where A i,j represents the attention values ​​at indices i and j, X represents the input (e.g., embedding of time series data 705), and W q T represents the query matrix, W k T represents the key matrix, C b_i and C b_j This is the first location code book that represents space, and T a_i and T a_j is a second position code book representing time, a and b represent indices, and T in the upper corner of the matrix indicates the mathematical operation of matrix transpose.

[0094] To eliminate higher-order cross-correlations (e.g., second-order correlations between time and space) resulting from processing based on Equation 2, these cross-correlations can be suppressed by pushing as many spatiotemporal calculations as possible to lower levels of processing without overloading. This provides an improvement over prior art, as conventional transformer models using conventional attention mechanisms require auxiliary structures to handle arbitrary relationships between time and space, and thus require additional computational resources. The relative distances in the time and spatial dimensions between segments of time-series data 705 can be encoded. Learnable bias terms u,v,w can be used to reduce the number of operations. Thus, Equation 2 can be extended to eliminate cross-terms representing interdimensional cross-correlations, resulting in Equation 3:A i,j =X i T Wq T W k_S X j +u T W k_S X j +X i T W q T W k_T T i-j +v T W k_T T i-j +X i T W q T W k_C C i-j +w T W k_C C i-j results in.

[0095] Equation 3 can be split into three components each representing a particular attention. For example, content-based attention is X i T W q T W k_S X j +u T W k_S X j which can be represented as quadratic in the number of inputs (e.g., O(T 2 C 2 ), where O represents complexity, T represents time, and C represents channel / space). Time-based attention is X i T W q T W k_T T i-j +v T W k_T T i-j which can be represented as sub-quadratic in the number of inputs (e.g., O(T 2 C)). Channel-based attention is X i T W q T W k_C C i-j +w T W k_C C i-jIt can be expressed as, and this is sub-quadratic with respect to the number of inputs (e.g., O(TC)). 2 Conventional attention mechanisms are fully quadratic, which represents a significant computational barrier, particularly in multivariate time series where handling inputs becomes difficult as the number of channels increases. At the same time, more channels suggest more sources of information, which cannot be ignored. In contrast, the time-based and channel-based attention components described herein are sub-quadratic, resulting in improved computational complexity, more efficient use of computational resources, and higher performance (e.g., 3.5 times faster processing time).

[0096] For example, if T is the number of time segments and C is the number of channels, then the context length of a conventional transformer-based model is T × C, and the number of terms that need to be calculated in conventional methods is O(T). 2 ×C 2 ). With a reasonable estimate of 100 segments and 50 channels, the context length becomes 5000, which can present a problem as it is difficult to handle even for language models. It should be understood that it is not necessary to compute a complete square matrix that is quadratic with respect to the context length (i.e., both time and space). All elements of time-based attention are the same with respect to each channel, and all elements of channel-based attention are the same with respect to each time point. Thus the complexity is quadratic in one dimension and constant in the other. Elements along a given dimension can be repeated without additional cost. Then, by employing a shift operation, all relative embeddings can be computed in a single pass.

[0097] The shift operation is configured to calculate the time-based attention and channel-based attention components. For example, in a time shift operation,

number

number

number

number

number

[0098] Channel shift operations can be performed as follows:

number

[0099] To further reduce the computational cost associated with calculating content-based attention with little impact on performance, a local attention window may be determined. The local attention window focuses on the most recent L time points and discards those with little informational content. Since time-based attention is unlimited, the lookup window still spans the entire context. Therefore, for L ≪ T, the total complexity of the self-attention mechanism 700A is O(T). 2 ×C+T×C 2 ) is quadratic in each dimension, but sub-quadratic with respect to context length. In certain embodiments, the self-attention mechanism 700A pushes the effective total context length up to over 10,000.

[0100] In certain embodiments, content-based attention can focus solely on the content of queries and keys without any positional encoding; time-based attention can focus solely on the temporal distance to queries and keys; and channel-based attention can focus solely on the spatial distance to queries and keys. Each attention component has its own key matrix to further increase semantic distance.

number

number

number

[0101] Figure 7B illustrates a pseudocode 700B that provides an exemplary method for implementing a multivariate parallel attention model (for example, by a multi-channel analysis code 200, a computer 101, etc.) according to one embodiment of the present invention. The algorithm of pseudocode 700B may take as input a plurality of output tokens generated by an encoder, where each output token x c,t This is multiple embeddings (for example, n embed Represented as an embedding within (=768). Each output token x c,t This is associated with channel encoding c and time encoding t. The algorithm for pseudocode 700B may include multiple attention heads (e.g., the total number of heads is parameter n). head (represented by ), and each attention head h is associated with query q. The algorithm of pseudocode 700B may include multiple grouped query attention (GQA) heads (for example, the total number of GQA heads is parameter n). gqa (represented by), each GQA head h k,vThis is associated with key k and value v. The algorithm for pseudocode 700B may also include biases u, v, and w.

[0102] The algorithm in pseudocode 700B may use the GQA mechanism. The GQA mechanism is configured to separate the calculation of queries from the calculation of keys and values. That is, for example, the output token x ct However, the query q is processed by the first linear operation without bias. ct h This is generated. Furthermore, the output token x ct This is processed by a second linear operation without bias to create the key.

number

number

number

number

number

[0103] To avoid recalculation, time-based and channel-based attention can be shifted (e.g., by the Transformer-XL model). For example, time-based attention s t h A time shift operation is performed on s tt' h This may be generated. Channel-based attention l c h A channel shift operation is performed on l cc' h This can be generated. Next, a causal mask can be applied to the sum of the three attention components. For example, if the causal mask is (g ct,c't' h +s tt' h +l cc' h ) Applied to output m ct,c't' h The following may be generated. Output m ct,c't' h This is further processed with a window mask operation and output n ct,c't' h The following is generated. Output n ct,c't' h A structured dropout operation is performed on the output d ct,c't' h The following can be generated. Output n ct,c't' h and parameter n embed The final attention value a is provided as input to the sigmoid function. ct,c't' h This is generated. Note the output. ct h is Σ c't' a ct,c't' h ·v ct,c't' h It is generated based on and returned as the result of the pseudocode 700B algorithm.

[0104] A comparison 700C between a structured dropout operation 752 (e.g., by a multi-channel analysis code 200, computer 101, etc.) and a conventional dropout operation 755, according to one embodiment of the present invention, is illustrated in Figure 7C. Dropout is a common technique for improving the generalization ability of neural networks. In transformer-based models, dropout is often applied within attention blocks, where some query-key attentions are randomly zeroed out to avoid the model becoming overly dependent on specific connections. Dropout is typically applied to all elements with equal probability, creating uniform holes in the attention matrix. However, this is not efficient in the case of multivariate time series, as neighboring segments are likely to carry very similar information for each hole, reducing the effectiveness of conventional dropout. For example, the conventional dropout operation 755 is configured to randomly blank out segments, which is ineffective for time series data because adjacent segments in time or space contain nearly the same information. To address these issues, the structured dropout operation 752 is configured to drop entire channels and / or entire time steps to reduce the number of correlated segments. The dropout rate may be calculated to maintain the same number of dropped segments as in the conventional dropout operation 755.

[0105] Structured dropout calculation 752 uses channel-specific dropout rate c drop and time-specific dropout rate t drop It may include c drop and t drop The calculation is as follows: the dropout rate r associated with the conventional dropout calculation 755 drop Based on:

number

[0106] Figure 7D illustrates a pseudocode 700D that provides an exemplary method for implementing a flash multivariate parallel attention model (flash MVPA) according to one embodiment of the present invention (e.g., by a multichannel analysis code 200, computer 101, etc.). The algorithm of pseudocode 700D may take as input a plurality of output tokens generated by an encoder, where each output token x ct This is multiple embeddings (for example, n embed Represented as an embedding within (=768). Each output token x ct This is associated with channel encoding c and time encoding t. The algorithm for pseudocode 700D may include multiple attention heads (e.g., the total number of heads is parameter n). head (represented by), and each attention head h is associated with query q. The algorithm of pseudocode 700D is to have multiple GQA heads (e.g., the total number of GQA heads is parameter n). gqa (represented by) may include each GQA head h k,v This is associated with key k and value v. The algorithm for pseudocode 700D may also include biases u, y, and w.

[0107] In the algorithm of pseudocode 700D, the GQA mechanism may be used. The GQA mechanism is configured to separate the calculation of queries from the calculation of keys and values. That is, for example, the output token x ct The query q is processed by the first linear operation without bias. ct h This is generated. Furthermore, the output token x ct This is processed by a second linear operation without bias to create the key.

number

number

number

number

[0108] The pseudocode 700D algorithm for implementing a flash multivariate parallel attention model can be leveraged to achieve efficient use of video random-access memory (VRAM) when performing classification and / or prediction tasks. The effectiveness of training a multivariate parallel attention model can be heavily influenced by the batch size, as its training process draws negative samples from batches. Larger batch sizes increase the diversity of negative samples and lead to better generalization of the model. In exemplary embodiments, considering large context sizes (e.g., up to 10k context sizes) for multivariate parallel attention models, a scaled dot product attention implementation can consume a significant amount of VRAM. Therefore, the pseudocode 700D algorithm can be leveraged to make VRAM consumption linear rather than quadratic with respect to context length, enabling training over much longer contexts. For example, the algorithm of pseudocode 700D can be implemented using an open-source programming language (e.g., the Triton language) that provides lower-level access to primitives of parallel computing platforms, such as Compute Unified Device Architecture (CUDA®) primitives. CUDA is a registered trademark of Nvidia Corporation. In certain embodiments, time-based and channel-based attention are computed by matrix multiplication operations in PyTorch, and the outputs of these computations are shifted and added by Triton. Content-based components can be fully implemented in Triton.

[0109] A method 800A according to one embodiment of the present invention, configured to perform a prediction task by a multivariate parallel attention model (e.g., by a multichannel analysis code 200, a computer 101, etc.), is illustrated in Figure 8A. First, a time series 805 may be processed in an encoding operation 810 (e.g., by an encoder 500A) to generate a plurality of embeddings 815. The time series 805 may be a multichannel time series (e.g., a multivariate time series) having a plurality of variables. For example, the time series 805 may be a dataset of iEEG signals collected in a clinical setting. The iEEG signal dataset may include recordings of neuronal activity of subjects showing both paroxysmal and non-paroxysmal events. Each window may be divided into segments, which may be processed to generate a plurality of tokens, which may be encoded into a plurality of embeddings 815.

[0110] Multiple embeddings 815 are provided as inputs to a multivariate parallel attention model 820. As described above, the multivariate parallel attention model 820 may be configured to determine time-based attention, channel-based attention, and content-based attention in parallel by a multivariate parallel attention mechanism. The multivariate parallel attention model 820 is then configured to perform a prediction task in a prediction operation 825. For example, the multivariate parallel attention model 820 is configured to generate multiple prediction outputs 830, which include multiple predicted future time series data corresponding to a time series 805. In a particular embodiment, the multivariate parallel attention model 820 may generate future neuronal activity based on recorded neuronal activity represented within the time series 805.

[0111] The time series 835 may be a multi-channel time series (e.g., a multivariate time series) with multiple variables. The time series 835 includes multiple observed time series data that follow the time series 805. For example, the time series 805 may include a record of neuronal activity from time 0 to time T, and the time series 835 may include a record of neuronal activity from time T+1 to another future time. The time series 835 may be processed in the encoding operation 840 (e.g., by an encoder) to generate multiple embeddings, which can be matched against multiple predicted outputs 830 in the matching operation 850. That is, multiple embeddings corresponding to the time series 835 (e.g., actual neuronal activity at time T+1) are compared against multiple predicted outputs 830 (e.g., neuronal activity at time T+1 predicted by the multivariate parallel attention model 820). The comparison may generate a loss, which can be optimized to train the multivariate parallel attention model 820.

[0112] A method 800B configured to train a multivariate parallel attention model (e.g., by a multi-channel analysis code 200, a computer 101, etc.) according to one embodiment of the present invention is illustrated in Figure 8B. Initially, a set of training data 860 is provided to train the multivariate parallel attention model 865. For example, the set of training data 860 may include an iEEG signal dataset collected in a clinical setting. The iEEG signal dataset may include recordings of neuronal activity of subjects exhibiting both paroxysmal and non-paroxysmal events. In certain embodiments, a sliding window technique may be used to increase the size of the training dataset and the number of training tokens by dividing each recording into overlap windows of a specific number of seconds (e.g., 500 seconds) with a specified percentage (e.g., 99%) of overlap. Different strides may be applied to paroxysmal and non-paroxysmal events to balance the dataset. The ratio between non-paroxysmal and paroxysmal events may be configured to obtain a balanced dataset (e.g., a 100:1 ratio would result in 1% of the data representing sejas). Each window is divided into segments, which are processed to generate multiple tokens, which can then be encoded into multiple embeddings. These multiple embeddings are provided as input to train a multivariate parallel attention model 865 to perform classification tasks (e.g., detecting paroxysmal events) and / or prediction tasks (e.g., generating neuronal activity).

[0113] In a particular embodiment, training of the multivariate parallel attention model 865 can be implemented in an end-to-end manner. In end-to-end training, B windows are randomly selected from multiple training data 860 to form a batch. Each window W i∈[1..B] This is any sample rate and C i It has n channels. The sampling rate can be normalized to a specified frequency (e.g., 512Hz), and then the window is divided into S non-overlapping segments per channel, resulting in C per window. i This results in ×S segments. Each segment passes through the encoder in parallel. For example, one window W *Assume that is randomly selected as a positive window, and all others are selected as confounding windows. * The embeddings form the input context E. In E, all embeddings corresponding to the last time step are removed, and as a result the context length is

number

[0114] For example, the multivariate parallel attention model 865 can be trained using control loss and auxiliary loss. It is important to have other windows within the batch to calculate the control loss, as larger batch sizes generally lead to more stable training and better generalization ability. For example, for a batch size of B, e i Let i∈[1,...,B] be the output of the signal encoder, d i Let i ∈ [1, ..., B] be the outputs of the decoder stack. * Regarding e i ,i≠i * n from negatives Each element is a negative sample

number

number

number

[0115] A classification head can be attached to the multivariate parallel attention model 865 to train it for classification tasks. The classification loss can be binary cross-entropy loss. The combination of the multivariate parallel attention model 865 and the classification head can be trained using Low-Rank Adaptation of Large Language Models (LoRA). For example, to improve the performance of the multivariate parallel attention model 865 on the task at hand, a small classification head can be created for each new task and dataset. The classification head consists of a single linear layer to keep computational overhead low. This layer has an input size equal to the block output size of the decoder and an output size equal to the dimensionality of the classification task (e.g., 2-dimensionality for a Seeja classification task). Input H to the classification head in This is the average pooled output of the last time series (e.g., signal) segment in time. For example,

number

[0116] In certain embodiments, the multivariate parallel attention model 865 may be trained over a predetermined period (e.g., two weeks) using multiple graphics processing units (GPUs) on a single node. An optimizer with a specified weight decay (e.g., FusedAdam with a weight decay of 0.1) may be selected to train the multivariate parallel attention model 865. The training strategy may be selected from a deep learning optimization library. The training strategy may not involve activation checkpointing, and the learning rate may be set to a specified value (e.g., 10). -4 ) can be fixed to . For example, compared to 1.2 hours per second per GPU for conventional attention models, the multivariate parallel attention model 865 can process 4.6 hours per second per GPU of data.

[0117] In an exemplary embodiment, the multivariate parallel attention model 865 can be trained to predict subsequent brain states using a dataset of intracranial EEG signals from patients suffering from epilepsy. The multivariate parallel attention model 865 can be trained with a large amount of data (e.g., 39 billion segments in the entire dataset, with 390 million unique segments) over a configurable time length (e.g., 5000 hours). The multivariate parallel attention model 865 can perform the task with or without fine-tuning (e.g., in a zero-shot manner). The multivariate parallel attention model 865 can reliably predict brain states during seizures with over 99% accuracy in predicting the next state from a set of 30 possible outputs. In addition to or concurrently with predicting the next brain state, the multivariate parallel attention model 865 can be trained to predict the occurrence of seizures and / or perform any type of classification related to the input iEEG data (e.g., sleep scoring, stroke detection, etc.). The performance of the multivariate parallel attention model 865 is superior to that of models fine-tuned for specific patients in the input data.

[0118] A method 800C according to one embodiment of the present invention, configured to perform a prediction task by a multivariate parallel attention model based on test data (e.g., by a multi-channel analysis code 200, a computer 101, etc.), is illustrated in Figure 8C. First, a set of test data 870 is provided to test the multivariate parallel attention model 875. For example, the set of test data 870 may include an iEEG signal dataset collected in a clinical setting. The set of test data 870 may be processed by an encoder to generate a set of embeddings. The multivariate parallel attention model 875 may be a model trained by a training procedure (e.g., method 800B). The multivariate parallel attention model 875 is configured to generate a set of outputs 880 (e.g., predictive outputs) based on a set of embeddings associated with the set of test data 870. The set of outputs 880 may be in the form of embeddings. For example, the multivariate parallel attention model 875 may predict future neuronal activity.

[0119] The generation of predictive outputs (e.g., brain signals representing neuronal activity) by the multivariate parallel attention model 875 can proceed in the same manner as during model training. Next, cosine similarity is measured directly in a three-way reference scheme. First, the cosine similarity between the output and the true target is determined. Second, the similarity between the output and the target with the highest correlation is determined. Third, the cosine similarity to random segments within the batch that are still temporally close with the best available form of entropy is determined. The cosine similarity measured by the three-way reference scheme ensures that the difference in similarity between the true target and the confounding target remains significant.

[0120] In certain embodiments, multiple outputs 880 (e.g., brain signals representing neuronal activity / events) may be further processed by post-processing operations. For example, post-processing operations include merging events that are within minutes and / or seconds (e.g., 5 minutes) of each other, removing events shorter than a specified time length (e.g., 20 seconds), and / or removing events with fewer than a specified number of positive responses (e.g., 5 positive responses). Furthermore, multiple sequences detected within 1 minute are merged into a single event. In addition, a threshold mechanism may be implemented to determine whether or not to report a sequence. For example, to suppress false positives, a threshold of 3 seconds being positive out of 10 seconds may be set as the lower limit for detecting a sequence. Events shorter than 3 seconds are not reported, and an additional 3 seconds of latency may be considered. Thresholds may be configured based on user requirements.

[0121] The performance of the multivariate parallel attention model 875 on classification and / or prediction tasks can be assessed based on multiple metrics. For example, for a seja detection task, the multivariate parallel attention model 875 can be evaluated based on a kappa score (also known as Cohen's kappa coefficient), which measures the degree of agreement between evaluators (e.g., between the model and experts) on the classification output generated by multiple evaluators. The kappa score ranges from 0 (no agreement) to 1 (perfect agreement). In the case of seja detection, there may be widespread disagreement among experts (e.g., neurologists) regarding atypical sejas, while at the same time there may be no disagreement at all regarding typical sejas. This phenomenon makes accurate classification difficult and contributes to the wide variation in performance between the model and human experts. To better assess the impact of this potential classification difficulty, multiple correlation analysis may be performed using three variables, including total record length, number of sejas, and frequency of sejas, to predict the kappa score. In certain embodiments, the multivariate parallel attention model 875 has a decision coefficient (R) of 0.054. 2 This can result in ). Therefore, model performance is independent of the three variables, and potential classification difficulties can help explain a large portion of the variance. Furthermore, compared to conventional transformer-based models, the multivariate parallel attention model 875 has a significantly higher (e.g., 1.9 times higher) Kappa score, indicating that the classification and / or predictive outputs from the multivariate parallel attention model 875 are closely aligned with expert classifications. In addition, performance metrics for the multivariate parallel attention model 875 that can be reported include F-1 score, sensitivity, false positive rate, etc. Compared to conventional transformer-based models, the multivariate parallel attention model 875 has improved performance metrics, including a lower false positive rate. Furthermore, in certain embodiments, the mean Kappa score of the multivariate parallel attention model 875 can be increased to 0.48, which is within the range of human expert performance.

[0122] Figure 8D illustrates a pseudocode 800D, which provides an exemplary method for performing an inference task by a multivariate parallel attention model (e.g., by a multi-channel analysis code 200, computer 101, etc.) according to one embodiment of the present invention. The input to the algorithm of pseudocode 800D is a plurality of raw inputs x (e.g., n segments This may include a time series having multiple segments, and each of the multiple unprocessed input segments (e.g., time series segments) x ij It is associated with a certain channel among multiple C channels and a specific time at time length T. Index i may be associated with a certain channel among the C channels, and index j may be associated with a specific time at time T. The input to the algorithm of pseudocode 800D is a question token q and the number of layers n layers It further includes n layers The parameters can be configured by the user.

[0123] In the algorithm of pseudocode 800D, multiple raw inputs x are segmented into multiple segments x through a segmentation operation. ij It is segmented into segments x. ij This is processed by the encoder. Separately, the question token q is also processed by the encoder. The encoder processes segment x ij And based on the question token q, embedded e ij The algorithm for pseudocode 800D generates the following: from layer l=1 to layer l=n layers The process proceeds iteratively until the decoder performs the embedded e ij Decode the decoded embedded e ij This is set as the answer embedding s. The generated embedding e i(j-1) The answer embeds are returned as the result of the algorithm of pseudocode 800D.

[0124] Embodiments of the present invention offer various technical and other advantages. For example, embodiments of the present invention utilize a transformer having a multivariate parallel attention model to generate prediction and / or classification of multi-channel input data. The multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel, thereby reducing the computational complexity associated with transformer-based analysis of multivariate time series. The multivariate parallel attention mechanism mitigates the problem of difficulty in handling data of varying lengths and dimensions, overcomes computational barriers, and results in more efficient use of computational resources and higher computational power. Furthermore, embodiments of the present invention provide a multivariate parallel attention model that can effectively capture contextual information by simultaneously paying attention to the time, channel, and content of the input data, thereby enabling improved performance of classification and / or prediction tasks.

[0125] It will be understood that the embodiments described above and illustrated in the drawings represent only a few of the many ways of implementing embodiments relating to multi-channel time series analysis with transformers accompanied by multivariate parallel attention models.

[0126] The environment of the embodiments of the present invention may include any number of computers or other processing systems (e.g., client or end-user systems, server systems, etc.) and databases or other repositories configured in any desired manner, wherein embodiments of the present invention may be applied to any desired type of computing environment (e.g., cloud computing, client-server, network computing, mainframe, standalone systems, etc.). The computers or other processing systems employed in embodiments of the present invention may be implemented by any number of any personal or other type of computers or processing systems. These systems may include any type of monitor and input device (e.g., keyboard, mouse, speech recognition, etc.) for inputting and / or viewing information.

[0127] The software of embodiments of the present invention (e.g., the multi-channel analysis code 200) can be implemented in any desired computer language and can be developed by those skilled in the computer art based on the functional descriptions contained herein and the flowcharts illustrated in the drawings. Furthermore, any reference herein that software performs various functions generally refers to a computer system or processor performing those functions under software control. The computer systems of embodiments of the present invention may be alternatively implemented by any type of hardware and / or other processing circuit configurations.

[0128] Various functions of a computer system or other processing system may be distributed in any manner among any number of software and / or hardware modules or units, processing systems or computer systems and / or circuit configurations, where the computer systems or processing systems may be located locally or remotely and may communicate via any suitable communication medium (e.g., LAN, WAN, intranet, internet, hardwired, modem connection, wireless, etc.). For example, the functions of embodiments of the present invention may be distributed in any manner among various end-user / client and server systems and / or any other intermediate processing devices. The software and / or algorithms described above and illustrated in the flowchart may be modified in any manner to realize the functions described herein. Furthermore, the functions in the flowchart or specification may be performed in any order to realize the desired operation.

[0129] The communication network may be implemented by any number of communication networks of any type (e.g., LAN, WAN, Internet, intranet, VPN, etc.). The computer or other processing system of the embodiment of the present invention may include conventional or other arbitrary communication devices for communicating over the network using conventional or other arbitrary protocols. The computer or other processing system may utilize any type of connection (e.g., wired, wireless, etc.) to access the network. The local communication medium may be implemented by any suitable communication medium (e.g., local area network (LAN), hardwired, wireless link, intranet, etc.).

[0130] The system may employ any number of any conventional or other databases, datastores, or storage structures (e.g., files, databases, data structures, data, or other repositories) to store information. The database system may be implemented by any number of any conventional or other databases, datastores, or storage structures (e.g., files, databases, data structures, data, or other repositories) to store information. The database system can be contained within or coupled to a server system and / or client system. The database system and / or storage structure can be remote from a computer or other processing system, or local to them, and can store any desired data.

[0131] Embodiments of the present invention may employ any number of user interfaces of any type (e.g., graphical user interfaces (GUIs), command lines, prompts, etc.) for acquiring or providing information (e.g., prediction and / or classification outputs, model parameters, etc.), and the interfaces may include any information arranged in any manner. The interfaces may include any number of input or actuation mechanisms of any type (e.g., buttons, icons, fields, boxes, links, etc.) placed anywhere to input / display information and to initiate desired actions via any suitable input device (e.g., mouse, keyboard, etc.). The interface screen may include any suitable actuators (e.g., links, tabs, etc.) for navigating between screens in any manner.

[0132] The report may contain any information arranged in any format and may be structured based on rules or other criteria for providing the user with desired information (e.g., prediction and / or classification outputs, model parameters, etc.).

[0133] Embodiments of the present invention are not limited to the specific tasks or algorithms described above, but can be used to generate prediction and / or classification outputs of any number of dimensions or channels of data from any data source.

[0134] The terminology used herein is intended to describe only specific embodiments and is not intended to limit the invention. Where used herein, unless otherwise explicitly indicated by the context, the singular forms "a," "an," and "the" are intended to include the plural forms. Where used herein, "comprises," "comprising," "includes," "including," "has," "have," "having," "with," and similar terms specify the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0135] In the following claims, all corresponding structures, materials, actions, and equivalents of means-plus-function elements or step-plus-function elements are intended to include any structures, materials, or actions for performing a function in combination with other specifically claimed elements.

[0136] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be comprehensive or limitless to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best describe the principles of the embodiments, their practical applications, or technical improvements to the technology available on the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for performing a prediction or classification task: The stage in which at least one processor generates multiple tokens from multiple multi-channel inputs; The steps of generating multiple embeddings from the multiple tokens using the encoder of at least one processor; and A step in which the multivariate parallel attention model of at least one processor performs the prediction task or the classification task based on the plurality of embeddings, wherein the multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel. A method that includes [a certain feature].

2. The steps for generating the multiple tokens, which have multi-channel time-series data, are as follows: The step of dividing the aforementioned multiple multichannel inputs into multiple time windows, each containing a subset of the aforementioned multiple multichannel inputs associated with time; The step of dividing each of the plurality of time windows into a plurality of segments, each associated with a channel represented within the plurality of multi-channel inputs; and A step of processing the multiple segments using a feature extraction technique to generate the multiple tokens. Having, The method according to claim 1.

3. The method according to claim 1, further comprising the step of training the multivariate parallel attention model with at least one processor to perform the prediction task and the classification task simultaneously, wherein the prediction task comprises predicting one or more data points following the plurality of multichannel inputs, and the classification task comprises classifying categories associated with the plurality of multichannel inputs.

4. The steps for training the aforementioned multivariate parallel attention model are: The step of determining the attention value for training the multivariate parallel attention model by summing the time-based attention, channel-based attention, and content-based attention. The method according to claim 3, having the following characteristics.

5. The method according to claim 1, wherein the multivariate parallel attention model includes a decoder and a multilayer perceptron.

6. The method according to claim 2, wherein the time-based attention is determined based on the temporal distance between each of the plurality of segments and between other segments of the plurality of segments, the channel-based attention is determined based on the spatial distance between each of the plurality of segments and between other segments, the content-based attention is determined by applying a self-attention mechanism to the plurality of segments, the temporal distance is stored in a temporal-spatial codebook, and the spatial distance is stored in a spatial-spatial codebook.

7. The method according to any one of claims 1 to 6, wherein one of the plurality of tokens is randomly selected as a question token representing the classification purpose, and one or more of the plurality of tokens are selected as answer tokens representing the classification output.

8. A computer system for performing a prediction task or a classification task, wherein: Processor set; One or more computer-readable storage media; and Stored on one or more computer-readable storage media, and to the processor set: Procedure for generating multiple tokens from multiple multi-channel inputs; A procedure for generating multiple embeddings from the multiple tokens using an encoder; and A procedure for performing the prediction task or the classification task based on the multiple embeddings using a multivariate parallel attention model, wherein the multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel. Program instructions that cause an operation to be performed A computer system equipped with the following features.

9. The aforementioned multiple multichannel inputs include multichannel time-series data, and the procedure for generating the aforementioned multiple tokens is: A procedure for dividing the aforementioned multiple multichannel inputs into multiple time windows, each containing a subset of the aforementioned multiple multichannel inputs associated with time; A procedure for dividing each of the aforementioned multiple time windows into a plurality of segments, each associated with a channel represented within the plurality of multi-channel inputs; and A procedure for processing the multiple segments using a feature extraction technique to generate the multiple tokens. The computer system according to claim 8, comprising:

10. The aforementioned operation is: The procedure further comprises training the multivariate parallel attention model to perform the prediction task and the classification task simultaneously, wherein the prediction task includes predicting one or more data points following the plurality of multichannel inputs, and the classification task includes classifying categories associated with the plurality of multichannel inputs. The computer system according to claim 8.

11. The procedure for training the aforementioned multivariate parallel attention model is as follows: A procedure for determining the attention value for training the multivariate parallel attention model by summing the time-based attention, channel-based attention, and content-based attention. The computer system according to claim 10, having the following features.

12. The computer system according to claim 8, wherein the multivariate parallel attention model includes a decoder and a multilayer perceptron.

13. The computer system according to claim 9, wherein the time-based attention is determined based on the temporal distance between each of the plurality of segments and between other segments of the plurality of segments, the channel-based attention is determined based on the spatial distance between each of the plurality of segments and between other segments, the content-based attention is determined by applying a self-attention mechanism to the plurality of segments, the temporal distance is stored in a temporal-spatial codebook, and the spatial distance is stored in a spatial-spatial codebook.

14. The computer system according to any one of claims 8 to 13, wherein one of the plurality of tokens is randomly selected as a question token representing a classification purpose, and one or more of the plurality of tokens are selected as answer tokens representing a classification output.

15. A computer program for performing a prediction task or a classification task, wherein the computer program is: Procedure for generating multiple tokens from multiple multi-channel inputs; A procedure for generating multiple embeddings from the multiple tokens using an encoder; and A procedure for performing the prediction task or the classification task based on the multiple embeddings using a multivariate parallel attention model, wherein the multivariate parallel attention model is configured to determine time-based attention, channel-based attention, and content-based attention in parallel. Program instructions that cause a set of processors to perform an operation having A computer program that includes the following features.

16. The aforementioned multiple multichannel inputs include multichannel time-series data, and the procedure for generating the aforementioned multiple tokens is: A procedure for dividing the aforementioned multiple multichannel inputs into multiple time windows, each containing a subset of the aforementioned multiple multichannel inputs associated with time; A procedure for dividing each of the aforementioned multiple time windows into a plurality of segments, each associated with a channel represented within the plurality of multi-channel inputs; and A procedure for processing the multiple segments using a feature extraction technique to generate the multiple tokens. The computer program according to claim 15, comprising:

17. The aforementioned operation is: The procedure further comprises training the multivariate parallel attention model to perform the prediction task and the classification task simultaneously, wherein the prediction task includes predicting one or more data points following the plurality of multichannel inputs, and the classification task includes classifying categories associated with the plurality of multichannel inputs. The computer program according to claim 15.

18. The procedure for training the aforementioned multivariate parallel attention model is as follows: A procedure for determining the attention value for training the multivariate parallel attention model by summing the time-based attention, channel-based attention, and content-based attention. A computer program according to claim 17, having the following characteristics.

19. The multivariate parallel attention model is a computer program according to any one of claims 15 to 18, comprising a decoder and a multilayer perceptron.

20. The computer program according to claim 16, wherein the time-based attention is determined based on the temporal distance between each of the plurality of segments and between other segments of the plurality of segments, the channel-based attention is determined based on the spatial distance between each of the plurality of segments and between other segments, the content-based attention is determined by applying a self-attention mechanism to the plurality of segments, the temporal distance is stored in a temporal-spatial codebook, and the spatial distance is stored in a spatial-spatial codebook.

Images