Block transformer architecture-based prediction method and system

Abstract

One embodiment provides a block transformer architecture-based prediction method performed by a computing device comprising a memory and a processor, the block transformer architecture-based prediction method comprising the steps of: processing input data in a sequence form so as to generate a plurality of input token embeddings; sequentially merging the plurality of input token embeddings into units of a predetermined number (N) so as to generate a plurality of block embeddings; performing a self-attention operation on the plurality of block embeddings so as to generate a plurality of context embeddings corresponding to the respective block embeddings; and generating subsequent prediction token embeddings for the plurality of input token embeddings on the basis of the plurality of context embeddings.

Description

Prediction method and system based on block transformer architecture

[0001] The present disclosure relates to a prediction method and system based on a block transformer architecture, and more particularly, to a prediction method and system based on a block transformer architecture, which can reduce computational load compared to a case where a self-attention operation is performed on all of a plurality of input token embeddings by converting a sequence input into a plurality of input token embeddings, grouping them into predetermined units to form a plurality of block embeddings, and calculating prediction data based on the results of performing a self-attention operation on the plurality of block embeddings.

[0002] With the recent advancement of artificial intelligence (AI) technology, various services utilizing AI are being commercialized across various industries. These AI technologies utilize artificial neural network models trained on vast amounts of data to output user-desired information from various input formats, such as images, voice, and text. This automates knowledge-intensive tasks like prediction, classification, and generation, rapidly improving productivity across industries.

[0003] Amidst this trend, large language models (LLMs) with tens of billions to hundreds of billions of learning parameters have emerged, dramatically improving natural language understanding and generation capabilities. LLMs are being applied across a wide range of fields, including question-answering, document summarization, multilingual translation, and code auto-completion, and are establishing themselves as core platforms in numerous industries, including search, education, healthcare, and fintech.

[0004] The Transformer, the foundation of LLM, is a query-key-value (QKV)-based attention mechanism that can learn relationships across input sequences in parallel. Transformers have evolved into encoder-decoder architectures, encoder-only architectures (BERT series), and decoder-only architectures (GPT series), and are widely used in diverse domains, including natural language processing, image processing, audio processing, and time-series data.

[0005] Among these, the decoder-only transformer sequentially generates subsequent tokens in an autoregressive manner, masking past tokens. This characteristic has led to its adoption as a mainstream model in applications sensitive to response delays, such as real-time conversational services and code completion.

[0006] However, since decoder-only transformers must fetch the key-value (KV) cache of all previous tokens at each step and compute the attention score, the computational load and memory I / O increase rapidly in proportion to the square of the token length L (O(L²)). Accordingly, although existing decoder-only transformers have excellent expressiveness when modeling long texts, they have a structural limitation in that the computational and memory costs increase excessively as the sequence length increases.

[0007] Therefore, research is needed on an improved transformer architecture that can significantly reduce computational load and memory consumption while maintaining the same prediction performance.

[0008] According to various embodiments of the present disclosure, a prediction method and system based on a block transformer architecture are provided, which can significantly reduce computational complexity and memory I / O by performing a global attention operation on a plurality of block embeddings generated by merging sequence inputs into predetermined units, and then applying the results of the global attention operation to a plurality of input token embedding groups corresponding to the plurality of block embeddings and then performing a local attention operation within each group.

[0009] According to various embodiments of the present disclosure, a prediction method and system based on a block transformer architecture are provided, which generates a plurality of context embeddings by performing a global attention operation on a plurality of block embeddings generated by merging a plurality of input tokens into a predetermined number of units, and applies each context embedding to a group of input token embeddings through a linear transformation, thereby more precisely reflecting global context information in a local attention operation performed within the group to output a prediction token.

[0010] However, the technical problems that the various embodiments of the present disclosure seek to solve are not limited to the technical problems described above, and other technical problems may exist.

[0011] One example is,

[0012] A block transformer architecture-based prediction method performed by a computing device including a memory and a processor, the method comprising: processing input data in a sequence form to generate a plurality of input token embeddings; merging the plurality of input token embeddings in sequence into a predetermined number of units (N) to generate a plurality of block embeddings; performing a self-attention operation on the plurality of block embeddings to generate a plurality of context embeddings corresponding to each block embedding; and generating a subsequent prediction token embedding for the plurality of input token embeddings based on the plurality of context embeddings.

[0013] In another aspect, the step of generating a subsequent predicted token embedding for the plurality of input token embeddings may include a step of sequentially generating a plurality of input token embeddings corresponding to subsequent block embeddings of a block embedding for the context embedding by inputting information about one of the plurality of context embeddings, and generating the subsequent token embeddings by referring to the generated previous token embeddings.

[0014] In another aspect, the block transformer architecture-based prediction method may further include a step of generating an additional subsequent block embedding after the last sequence by merging the plurality of input token embeddings generated based on a context embedding corresponding to the last sequence among the plurality of context embeddings, and a step of generating a plurality of context embeddings corresponding to each block embedding by performing a self-attention operation again on the plurality of block embeddings and the additional subsequent block embeddings.

[0015] In another aspect, in the step of generating the subsequent token embedding, a plurality of input token embeddings corresponding to subsequent block embeddings of block embeddings for the context embeddings can be sequentially generated for all of the plurality of context embeddings.

[0016] In another aspect, each of the plurality of block embeddings can be generated by concatenating a plurality of input token embeddings of the predetermined number of units (N) arranged in order.

[0017] In another aspect, the above-described number of units (N) may be 4.

[0018] In another aspect, the step of generating the subsequent token embedding based on any one of the context embeddings may include the step of generating at least one context-embedded embedding based on any one of the context embeddings, and the step of generating the subsequent token embedding in a self-regressive manner based on a self-attention operation on the at least one context-embedded embedding and the sequentially generated previous token embedding.

[0019] In another aspect, in the step of generating the at least one context-embedded embedding, the at least one context-embedded embedding can be generated through a linear transformation of any one of the context embeddings.

[0020] In another aspect, in the step of generating at least one context embedding, a plurality of context embeddings can be generated through linear transformation of any one of the context embeddings.

[0021] In another aspect, learning parameters may be evenly distributed to a block decoder that performs self-attention operations on the plurality of block embeddings and a token decoder that performs self-attention operations on the at least one context-embedded embedding and the sequentially generated previous token embeddings.

[0022] In another aspect, the step of generating the plurality of input token embeddings may include the step of processing input data in the form of a sequence to generate the plurality of input tokens, and the step of generating the plurality of input token embeddings based on the plurality of input tokens.

[0023] One example is,

[0024] A block transformer architecture-based prediction system is provided, comprising at least one memory and at least one processor that reads at least one instruction stored in the memory to perform a transformer architecture-based prediction method, wherein the at least one processor processes input data in a sequence form to generate a plurality of input token embeddings, sequentially merges the plurality of input token embeddings into a predetermined number of units (N) to generate a plurality of block embeddings, performs a self-attention operation on the plurality of block embeddings to generate a plurality of context embeddings corresponding to each block embedding, and generates a subsequent prediction token embedding for the plurality of input token embeddings based on the plurality of context embeddings.

[0025] In another aspect, the at least one processor may sequentially generate a plurality of input token embeddings corresponding to a subsequent block embedding of a block embedding for the context embedding based on the plurality of context embeddings, by inputting information about one context embedding among the plurality of context embeddings, and may generate the subsequent token embedding by referencing the generated previous token embedding.

[0026] In another aspect, the block transformer architecture-based prediction system may further include a field programmable gate array (FPGA) implementation for a predetermined artificial neural network, including a plurality of neurons configured as an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; and at least one routing network that controls data flow between the plurality of neurons, wherein each of the plurality of neurons is connected to at least one other neuron through the routing network to set a transmission path of the weights.

[0027] In another aspect, the block transformer architecture-based prediction system may further include a plurality of neurons organized in an array including at least one register, at least one microprocessor, and at least one input; and a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons, wherein each of the plurality of neurons may further include an Application Specific Integrated Circuit (ASIC) for a given artificial neural network connected to at least one other neuron through one of the plurality of synaptic circuits.

[0028] The block transformer architecture-based prediction method and system according to various embodiments of the present disclosure can enable extraction of block-level macroscopic contextual information for a sequence input by performing a global attention operation on a plurality of block embeddings generated by merging sequence inputs into predetermined units.

[0029] In addition, the prediction method and system based on the block transformer architecture according to various embodiments of the present disclosure outputs a predicted token embedding by applying the result of a global attention operation to a plurality of input token embedding groups corresponding to a plurality of block embeddings and then performing a local attention operation within each group, thereby significantly reducing computational complexity and memory I / O, thereby maintaining high inference performance even in the process of processing a long sequence input with a significantly lower amount of computation and memory requirements compared to the existing transformer architecture.

[0030] Furthermore, the prediction method and system based on the block transformer architecture according to various embodiments of the present disclosure perform a global attention operation on a plurality of block embeddings generated by merging a plurality of input tokens into a predetermined number of units, and apply the plurality of context embeddings generated thereby to a plurality of input token embedding groups through linear transformation, thereby enabling global context information to be precisely reflected in the local attention operation performed within each group, and accordingly, the meaning of the entire sequence input can be appropriately interpreted and the corresponding prediction token can be output more accurately.

[0031] However, the effects that can be obtained through various embodiments of the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood from the description below.

[0032] FIG. 1 illustrates an example block diagram of a computing system that implements a prediction service based on a block transformer architecture according to one embodiment.

[0033] FIG. 2 is a schematic diagram illustrating the structure of a neuromorphic circuit that a processor may include according to one embodiment.

[0034] FIG. 3 illustrates an example block diagram of a computing device implementing a prediction service based on a block transformer architecture according to one embodiment.

[0035] FIG. 4 illustrates an example block diagram from another aspect of a computing device implementing a prediction service based on a block transformer architecture according to one embodiment.

[0036] FIG. 5 is a block diagram illustrating an exemplary configuration of a block transformer architecture according to one embodiment.

[0037] FIG. 6 is for explaining the configuration of various layers included in a block transformer architecture according to one embodiment.

[0038] FIG. 7 is a flowchart illustrating a method for generating at least one context-embedded embedding according to one embodiment.

[0039] FIG. 8 is a graph showing the learning performance of a language model according to a method of reflecting multiple context embeddings to a token decoder in a block transformer architecture according to one embodiment.

[0040] Figure 9 is a graph comparing the change in perplexity of a language model according to the length of the block embedding and the distribution ratio of learning parameters between the block decoder and the token decoder.

[0041] Fig. 10 is a flowchart of a prediction method based on a block transformer architecture according to one embodiment.

[0042] FIG. 11 is a flowchart of a step of generating a subsequent token embedding based on any one of the context embeddings included in the block transformer architecture-based prediction method of FIG. 10.

[0043] The present invention is capable of various modifications and embodiments. Therefore, specific embodiments are illustrated in the drawings and described in detail in the detailed description. The effects and features of the present invention, as well as the methods for achieving them, will become clear with reference to the embodiments described in detail below together with the drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various forms. In the following embodiments, the terms "first," "second," etc. are not used in a limiting sense but are used for the purpose of distinguishing one component from another. Furthermore, the singular expression includes the plural expression unless the context clearly indicates otherwise. Furthermore, terms such as "include" or "have" indicate the presence of a feature or component described in the specification, and do not preemptively exclude the possibility that one or more other features or components may be added. Furthermore, the sizes of components in the drawings may be exaggerated or reduced for convenience of explanation. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of explanation, and thus the present invention is not necessarily limited to what is shown.

[0044] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. When describing with reference to the drawings, identical or corresponding components are given the same drawing reference numerals, and redundant descriptions thereof will be omitted.

[0045]

[0046] - System (1000) providing prediction service based on block transformer architecture

[0047] A system (1000) according to one embodiment processes input data in the form of a sequence to generate a plurality of tokens, performs a global attention operation on a plurality of block embeddings generated by merging them into a predetermined number of units (N), and then applies the result of the global attention operation to a plurality of input token groups corresponding to the plurality of block embeddings, and then performs a local attention operation within each group to generate a predicted token in an autoregressive manner.

[0048] In this way, unlike the conventional general transformer architecture having O(L²) computational complexity for all L input tokens, the block transformer architecture of the system (1000) according to one embodiment performs attention operations with O(N²) complexity for each of L / N groups, thereby learning a global context for input data with a smaller computational complexity of O((L / N)·N²=L·N) level in total, and can generate an appropriate prediction token for the input data based on the attention operations.

[0049] In addition, the KV-cache size and memory I / O required for attention operations in each layer of the block transformer architecture of the system (1000) according to one embodiment can be reduced to the level of 1 / N to 1 / N², thereby significantly improving both batch throughput and latency during inference. Thanks to these structural characteristics, the system (1000) according to one embodiment can rapidly generate prediction data while significantly improving computational efficiency and memory efficiency even in long sequences.

[0050] FIG. 1 illustrates an example of a block diagram of a computing system (1000) that implements a prediction service based on a block transformer architecture according to one embodiment.

[0051] Referring to FIG. 1, a computing system (1000) implementing a prediction service based on a block transformer architecture according to one embodiment includes a user computing device (110), a server computing system (130), and a training computing system (150), which are capable of communicating with each other via a network (170).

[0052] A prediction method based on a block transformer architecture according to one embodiment may be implemented and provided locally by a user computing device (110), implemented and provided in the form of a web service by a server computing system (130) communicating with the user computing device (110), or implemented and provided by a user computing device (110) and a server computing system (130) in connection with each other.

[0053] At this time, in the embodiment, the user computing device (110) and / or the server computing system (130) can train the machine learning model (120 and / or 140) through interaction with a training computing system (150) communicatively connected via a network (170). The training computing system (150) may be separate from the server computing system (130) or may be part of the server computing system (130).

[0054] And at this time, the artificial intelligence model can be 1) trained directly locally by the user computing device (110), 2) trained by the server computing system (130) and the user computing device (110) interacting with each other through a network (170), and 3) trained by a separate training computing system (150) using various training techniques and learning techniques. And the artificial intelligence model trained by the training computing system (150) can be implemented in a manner of being provided / updated by transmitting the model to the user computing device (110) and / or the server computing system (130) through a network (170).

[0055] In some embodiments, the training computing system (150) may be part of a server computing system (130) or part of a user computing device (110).

[0056] -User Computing Device (110)

[0057] The user computing device (110) may include any type of computing device, such as a smart phone, a mobile phone, a digital broadcasting device, a personal digital assistant (PDA), a portable multimedia player (PMP), a desktop, a wearable device, an embedded computing device, and / or a tablet PC.

[0058] Additionally, in the embodiment, the user computing device (110) may further include a predetermined server computing device that provides a prediction service environment based on a block transformer architecture.

[0059] Such a user computing device (110) includes at least one processor (111) and memory (112).

[0060] Here, the processor (111) of the user computing device (110) may be composed of at least one or a plurality of processors electrically connected among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors and / or other electrical units for performing functions.

[0061] In particular, depending on the embodiment, such a processor (111) may be configured based on a field programmable gate array (FPGA) implementation, which is a hardware technology for implementing a predetermined digital circuit, and / or an application specific integrated circuit (ASIC).

[0062] Here, a field programmable gate array (FPGA) can mean a flexible digital circuit that can be programmed according to user needs.

[0063] In an embodiment, the field programmable gate array implementation may include a register that temporarily stores data and controls the flow and timing of signals to maintain intermediate results of operations or state information to support synchronized operation of the FPGA; programmable logic that programs operations within the FPGA to perform specific functions or operations as logic circuits that can be configured according to user needs; and an input interface that receives signals from an external device or sensor as a path for receiving data from outside the FPGA and transmits them to the internal circuit.

[0064] By combining the above components, field programmable gate array implementations can provide flexible and diverse forms of digital circuits.

[0065] Meanwhile, an application-specific integrated circuit (ASIC) can refer to a custom integrated circuit that is specifically designed to perform a specific purpose or function.

[0066] As an example, the application-specific integrated circuit may include a register, which is a small memory device that temporarily stores and manages data and stores intermediate results of calculations or status information to support rapid processing of the operation of the ASIC; a microprocessor, which is a central processing unit that performs control and operations within the ASIC and, if necessary, performs various operations or generates control signals to coordinate the operation of the entire system; and an input block, which is an interface that receives data from the outside and receives data to be processed by the ASIC and transmits it to the inside, and receives various input data through a connection with a sensor or an external device.

[0067] By combining the above components, application-specific integrated circuits can perform specific tasks in an optimized manner.

[0068] For example, an ASIC may have a structure of a neuromorphic circuit in the form of an array containing multiple neuron circuits.

[0069] FIG. 2 is a schematic diagram illustrating the structure of a neuromorphic circuit that a processor (111) may include according to one embodiment.

[0070] Referring to FIG. 2, for example, a neuromorphic circuit (300) may include a plurality of presynaptic neuron circuits (310), a plurality of presynaptic lines (311) extending laterally from the plurality of presynaptic neuron circuits (310), a plurality of post-synaptic neuron circuits (320), a plurality of post-synaptic lines (321) extending longitudinally from the plurality of post-synaptic neuron circuits (320), and a plurality of synaptic circuits (330) provided at intersections of the plurality of presynaptic lines (311) and the plurality of post-synaptic lines (321).

[0071] A plurality of presynaptic neuron circuits (310) can transmit signals input from the outside in the form of electrical signals to a plurality of synaptic circuits (330) through a plurality of presynaptic lines (311).

[0072] Additionally, a plurality of post-synaptic neuron circuits (320) can receive electrical signals from a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0073] Furthermore, a plurality of post-synaptic neuron circuits (320) may transmit electrical signals to a plurality of synaptic circuits (330) through a plurality of post-synaptic lines (321).

[0074] A plurality of synaptic circuits (330) can store weights included in layers constituting a neural network system implemented by a neuromorphic circuit (300) and perform a predetermined operation based on the weights and input data.

[0075] For example, each of the plurality of synaptic circuits (330) may include a resistive memory cell having a variable resistance. In this case, the plurality of synaptic circuits (330) may have a resistance value that changes according to a voltage applied through the plurality of presynaptic neuron circuits (310) or the plurality of postsynaptic neuron circuits (320), and may store weight data according to this resistance change.

[0076] The neuromorphic circuit (300) is formed by simulating the neuron and synapse structures, which are essential elements of the human brain. When a deep neural network (DNN) is implemented using the neuromorphic circuit (300), data processing speed can be improved and power consumption can be reduced compared to when the existing von Neumann structure is utilized.

[0077] Returning again, the memory (112) of the user computing device (110) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof, and may include web storage of a server that performs a memory storage function on the Internet. The memory (112) may store data (113) and instructions (114) necessary for performing functional operations such as allowing the at least one processor (111) to train an artificial intelligence model, process a sequence input through an artificial intelligence model to generate a plurality of input token embeddings, or perform a global attention operation on a plurality of block embeddings generated by merging a plurality of input token embeddings into a predetermined number of units (N), and / or perform a local attention operation within a plurality of input token embedding groups corresponding to the plurality of block embeddings.

[0078] In one embodiment, the user computing device (110) can perform various deep learning for a block transformer architecture-based prediction service in conjunction with a deep learning neural network.

[0079] Here, the deep learning neural network according to the embodiment may include a convolutional neural network (CNN), an R-CNN (Regions with CNN features), a Fast R-CNN, a Faster R-CNN, a Mask R-CNN, etc., and may include any deep learning neural network that includes an algorithm capable of performing the embodiment described below, and the embodiment of the present invention does not limit or restrict such deep learning neural network itself.

[0080] At this time, depending on the embodiment, the deep learning neural network may be installed directly in the server computing system (130) or may operate as a separate device from the server computing system (130) to perform deep learning for a prediction service based on a block transformer architecture.

[0081] Additionally, in one embodiment, the user computing device (110) may store at least one machine learning model (120).

[0082] For example, the user computing device (110) may be configured with a combination of various machine learning models, such as multiple neural networks (e.g., deep neural networks) that perform a block transformer architecture-based prediction method based on structured / quantitative data, or other types of machine learning models including nonlinear models and / or linear models.

[0083] For example, the machine learning model may store linear regression, decision tree, random forest, gradient boosting, pre-trained language model, or / and deep learning model. The neural network may include at least one of feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or / and other types of neural networks.

[0084] Additionally, according to an embodiment, the user computing device (110) may store a model to be used in each process and a prompt template that serves as the basis for input to the model in order to perform at least part of the process performed for a block transformer architecture-based prediction method through a large-scale language model (LLM).

[0085] In one embodiment, the user computing device (110) receives at least one machine learning model (120) from a server computing system (130) via a network (170), stores the received machine learning model (120) in a memory (112), and then executes the stored machine learning model (120) via a processor (111) to process a sequence input to generate a plurality of input token embeddings, or perform a global attention operation on a plurality of block embeddings generated by merging a plurality of input token embeddings into a predetermined number of units (N), and / or a local attention operation within a plurality of input token embedding groups corresponding to the plurality of block embeddings.

[0086] In another embodiment, a user computing device (110) may perform an operation through a machine learning model (140) including at least one machine learning model (140) in conjunction with a server computing system (130) and provide a block transformer architecture-based prediction service to a user by communicating data related thereto to the outside.

[0087] For example, a user computing device (110) can perform a block transformer architecture-based prediction service in a manner that a server computing system (130) provides output for a user's input using a machine learning model (140) via the web.

[0088] Additionally, the artificial intelligence model may be implemented in such a way that at least some of the machine learning models (120 and / or 140) are executed on a user computing device (110) and the rest are executed on a server computing system (130).

[0089] Additionally, the user computing device (110) may include at least one input component (121) that detects user input.

[0090] For example, the user input component (121) may include a touch sensor (e.g., a touch screen and / or touch pad, etc.) that detects the touch of a user's input medium (e.g., a finger or a stylus), an image sensor that detects the user's motion input, a microphone, button, mouse, and / or keyboard that detects the user's voice input, etc.

[0091] Here, the image sensor may include an image processing module. Specifically, the image sensor may process still images or moving images obtained by an image sensor device (e.g., CMOS or CCD).

[0092] Additionally, the image sensor can process still images or moving images acquired through the image sensor device using an image recognition process (e.g., OCR, etc.) and / or an image processing module to extract necessary information and transmit the extracted information to the processor.

[0093] Additionally, the input component (121) may receive input for an external controller (e.g., mouse, keyboard, etc.) based on the interface module, and in this case may include an external output device (e.g., speaker).

[0094] At this time, the interface module may be configured to include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, a power amplifier, an RF circuit, a transceiver, and other communication circuits.

[0095] Additionally, the external output device may include a display system that outputs various information related to a block transformer architecture-based prediction service as a graphic image.

[0096] Such a display system can be implemented by including at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display, a 3D display, and an e-ink display.

[0097] Meanwhile, the user computing device (110) including the above-described components may further perform at least some of the functional operations performed by the server computing system (130) described below.

[0098] -Server Computing System (130)

[0099] The server computing system (130) can perform a series of processes to provide a prediction service based on a block transformer architecture.

[0100] In detail, in an embodiment, the server computing system (130) can provide the block transformer architecture-based prediction service by exchanging data necessary to enable the block transformer architecture-based prediction service process to be driven by an external device, such as a user computing device (110), with the external device.

[0101] In more detail, in an embodiment, a server computing system (130) may provide an environment in which an application can operate on a user computing device (110).

[0102] To this end, the server computing system (130) may include application programs, data and / or commands for the application to operate, and may transmit and receive various data based thereon with the external device.

[0103] Additionally, the server computing system (130) includes at least one processor (131) and memory (132).

[0104] Here, the processor (131) of the server computing system (130) may be composed of at least one or a plurality of processors electrically connected among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors and / or other electrical units for performing functions.

[0105] In particular, depending on the embodiment, such a processor (131) may be configured based on a Field Programmable Gate Array (FPGA) implementation, which is a hardware technology for implementing a predetermined digital circuit, and / or an Application Specific Integrated Circuit (ASIC). A detailed description thereof is omitted in accordance with the description of the FPGA and ASIC described above.

[0106] And the memory (132) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory (132) may store data (133) and instructions (134) necessary for the processor (131) to perform functional operations such as training an artificial intelligence model, processing a sequence input through an artificial intelligence model to generate a plurality of input token embeddings, performing a global attention operation on a plurality of block embeddings generated by merging a plurality of input token embeddings into a predetermined number of units (N), and / or performing a local attention operation within a plurality of input token embedding groups corresponding to the plurality of block embeddings.

[0107] In one embodiment, the server computing system (130) may be implemented to include at least one computing device. For example, the server computing system (130) may be implemented to operate multiple computing devices according to a sequential computing architecture, a parallel computing architecture, or a combination thereof. Additionally, the server computing system (130) may include multiple computing devices connected via a network (170).

[0108] Additionally, the server computing system (130) may store at least one machine learning model (140). For example, the server computing system (130) may include a neural network and / or other multi-layer nonlinear model as the machine learning model (140). Exemplary neural networks may include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0109] In an embodiment, the server computing system (130) may further include a data store computing system (hereinafter, “data store”), which is a storage for continuously storing and managing raw data that is the basis of a block transformer architecture-based prediction service.

[0110] These data stores can include various forms of data storage, ranging from file systems to cloud storage. For example, a data store can include at least one of the following: a relational database that uses a structured query language (SQL) to define and manipulate data; a NoSQL database designed for flexibility and scalability to handle unstructured and semi-structured data; a data warehouse, which is a system used for reporting and data analysis, centralizes large amounts of data from multiple sources and is optimized for querying and analysis; a data warehouse that stores large amounts of raw data in its native formats of structured, semi-structured, and unstructured data; and a database on a local storage device or Network Attached Storage (NAS) that stores data in files in a format typically accessible by a computer operating system.

[0111] -Training Computing System (150)

[0112] The training computing system (150) includes at least one processor (151) and memory (152).

[0113] Here, the processor (151) of the training computing system (150) may be composed of at least one or a plurality of processors electrically connected among a central processing unit (CPU), a graphics processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors and / or other electrical units for performing functions.

[0114] In particular, depending on the embodiment, such a processor (151) may be configured based on a Field Programmable Gate Array (FPGA) implementation, which is a hardware technology for implementing a predetermined digital circuit, and / or an Application Specific Integrated Circuit (ASIC). A detailed description thereof is omitted in accordance with the description of the FPGA and ASIC described above.

[0115] And the memory (152) may include one or more non-transitory / transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. This memory (152) may store data (153) and instructions (154) necessary for the processor (151) to perform learning of an artificial intelligence model, etc.

[0116] For example, the training computing system (150) may include a model trainer (160) that trains a machine learning model (120 and / or 140) stored in the user computing device (110) and / or server computing system (130) using various training or learning techniques, such as backpropagation of errors (according to the framework illustrated in FIG. 4).

[0117] For example, such a model trainer (160) may perform updates for one or more parameters of a machine learning model (120 and / or 140) for a block transformer architecture-based prediction service in a backpropagation manner based on a defined loss function.

[0118] In some implementations, performing backward propagation of errors may include performing truncated backpropagation through time. The model trainer (160) may perform a number of generalization techniques (e.g., weight reduction, dropout, and / or knowledge distillation) to improve the generalization ability of the trained machine learning model (120 and / or 140).

[0119] Additionally, the model trainer (160) may train the machine learning model (120 and / or 140) based on a series of training data (161). Here, the training data (161) may include data in different formats, such as, for example, images, audio samples, and / or text. Examples of image types that may be used may include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0120] Such training data (161) may be provided by a user computing device (110) and / or a server computing system (130). When the training computing device trains a machine learning model (120 and / or 140) on specific data of the user computing device (110), the machine learning model (120 and / or 140) may be characterized as a personalized model.

[0121] And the model trainer (160) includes computer logic utilized to provide the desired function.

[0122] Additionally, the model trainer (160) may be implemented as hardware, firmware, and / or software that controls a general-purpose processor. In one implementation, the model trainer (160) includes a program file stored in a storage device, which may be loaded into a memory (152) and executed by one or more processors (151). In another implementation, the model trainer (160) includes one or more sets of computer-executable data (153) and instructions (154) stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0123] The network (170) includes, but is not limited to, a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), a Wireless Local Area Network (Wireless LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and / or a Digital Multimedia Broadcasting (DMB) network.

[0124] In general, communication over a network (170) may be performed using any type of wired and / or wireless connection, using various communication protocols (e.g., TCP / IP, HTTP, SMTP, and / or FTP), encodings or formats (e.g., HTML and / or XML), and / or protection schemes (e.g., VPN, Secure HTTP, and / or SSL).

[0125] FIG. 3 illustrates an example of a block diagram of a computing device (100) that implements a prediction service based on a block transformer architecture according to one embodiment.

[0126] Referring to FIG. 3, the computing device (100) included in the user computing device (110), the server computing system (130), and the training computing system (150) includes a plurality of applications (e.g., Application 1 to Application N). Each application may include a machine learning library and one or more machine learning models. For example, the applications may include an image processing (e.g., Detection, Classification, and / or Segmentation, etc.) application, a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and / or a chat-bot application.

[0127] In an embodiment, the computing device (100) may include a model trainer (160) for training an artificial intelligence model, and may store and operate the trained artificial intelligence model to provide output data according to predetermined input data (e.g., a conversation data set, etc.).

[0128] Each application of the computing device (100) may communicate with a number of other components of the computing device (100), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In one embodiment, each application may communicate with each device component using an API (e.g., a public API). In one embodiment, the API used by each application may be specific to that application.

[0129] FIG. 4 illustrates an example block diagram from another aspect of a computing device (200) implementing a prediction service based on a block transformer architecture according to one embodiment.

[0130] Referring to FIG. 4, a computing device (200) includes a plurality of applications (e.g., Application 1 to Application N). Each application may communicate with a central intelligence layer. For example, the applications may include an image processing application, a text messaging application, an email application, a dictation application, a virtual keyboard application, and / or a browser application. In one embodiment, each application may communicate with the central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0131] The central intelligence layer may include multiple machine learning models. For example, as illustrated in FIG. 4, at least some of the machine learning models may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model to all applications. In some implementations, the central intelligence layer may be included within the operating system of the computing device (200) or implemented differently.

[0132] The central intelligence layer may communicate with a central device data layer. The central device data layer may be a centralized data repository for the computing device (200). As illustrated in FIG. 5, the central device data layer may communicate with a number of other components of the computing device (200), such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer may communicate with each device component using an API (e.g., a private API).

[0133] The techniques described herein may refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information transmitted to or from such systems. It will be appreciated that the inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and division of labor and functionality between and among components. For example, the processes described herein may be implemented using a single device or component, or multiple devices or components operating in combination. Databases and applications may be implemented on a single system or in a distributed system across multiple systems. Distributed components may operate sequentially or in parallel.

[0134] FIG. 5 is a block diagram illustrating an exemplary configuration of a block transformer architecture (180) according to one embodiment.

[0135] Referring to FIG. 5, a block transformer architecture (180) used to perform a block transformer architecture-based prediction method according to various embodiments of the present disclosure may include a block embedding generation layer (10), a context embedding generation layer (20), and a prediction token embedding generation layer (30).

[0136] When input data (ID) is input from the outside into the block transformer architecture (180), the block embedding generation layer (10) can process the input data (ID) to generate multiple block embeddings.

[0137] Input data (ID) in the form of a sequence can be received by the processor (111, 131, 151). For example, a user can provide the input data in the form of a natural language command, question, file, sensor data, etc. through a user computing device (110).

[0138] In this case, the input data (ID) can be of various types, including text, speech, image patches, video frames, sensor time series data, radar / lidar signals, code snippets, protein molecular structures, and behavioral logs. This data is converted into sequence form and used as input for subsequent processing.

[0139] Input data (ID) can be tokenized by the processor (111, 131, 151). For example, text data can be segmented into words or subwords, and speech data can be segmented into time-frequency patches based on MFCCs, Mel Spectrograms, or Fourier transform coefficients. Image or video inputs are segmented into fixed-size patches and used as tokens.

[0140] The multiple input tokens generated as a result of tokenization of input data (ID) can be converted into multiple input token embeddings by the processor (111, 131, 151). For example, each token is mapped to a vector space of a predetermined dimension, and the corresponding embeddings can be generated by a pre-trained embedding table or a convolution-based or linear projection-based embedding layer.

[0141] The block embedding generation layer (10) can generate multiple block embeddings by sequentially merging multiple input token embeddings into a predetermined number of units (N). This merging process is a preprocessing step that divides the entire sequence into blocks of a certain size to increase the efficiency of subsequent attention operations.

[0142] For example, token embeddings of a given number of consecutive units (N) can be combined into a single block embedding by simple concatenation or mean pooling. These block embeddings can then be used as input for block-level global attention operations.

[0143] The context embedding generation layer (20) can perform a global attention operation on a plurality of block embeddings to generate a plurality of context embeddings that include global context information reflecting the relationship between the plurality of block embeddings.

[0144] The context embedding generation layer (20) may use a block decoder structure in the form of a transformer decoder to receive multiple block embeddings as input and model the relationships between them. The block decoder may learn what semantic relationships each block has with its previous blocks by performing a masked self-attention operation between the multiple block embeddings, and may output a context embedding corresponding to each block from the result of this operation.

[0145] At this point, applying masked self-attention ensures an autoregressive flow during the computation, which is crucial for maintaining causality during subsequent token predictions. Each context embedding serves as a summary vector of global context information extracted from the corresponding block embedding and previous block embeddings, and can serve as the basis for subsequent local attention-based token prediction operations.

[0146] The context embedding generation layer (20) can be implemented in a form in which multiple transformer decoder layers are stacked, and each layer uses block embedding to generate a query (Q), a key (K), and a value (V), and transmits global information through mutual attention weights between each block.

[0147] In this way, the context embedding generation layer (20) can be said to be a key component that minimizes context disconnection between blocks, maintains global semantic coherence, and enables a balance between computational efficiency and expressive power for long sequence inputs under a structure that divides the entire sequence of input data into block units and processes it.

[0148] The prediction token embedding generation layer (30) can generate a prediction token embedding in an autoregressive manner by taking information from one of a plurality of context embeddings as input, applying it to an input token group corresponding to a subsequent block embedding of a block embedding for the context embedding, and then performing a local attention operation within the token group.

[0149] Additionally, prediction data (PD) for input data in the form of a sequence can be provided based on the prediction token embedding generated by the prediction token embedding generation layer (30).

[0150] In this way, global context information for sequence input is acquired as a global attention operation is performed in the context embedding generation layer (20), and a predicted token embedding can be generated as a local attention operation is repeatedly performed for each of a plurality of input token groups in the predicted token embedding generation layer (30).

[0151] Accordingly, the block transformer architecture (180) according to one embodiment can maintain high inference performance with significantly lower computational complexity and memory I / O compared to existing transformer architectures even in the process of processing long sequence inputs.

[0152] Hereinafter, the configuration and functions of various layers included in a block transformer architecture (180) according to one embodiment will be described with reference to FIGS. 6 and 7.

[0153] FIG. 6 is a diagram illustrating the configuration of various layers included in a block transformer architecture (180) according to one embodiment. FIG. 7 is a diagram illustrating a method for generating at least one context-embedded embedding according to one embodiment.

[0154] The block transformer architecture (180) illustrated in FIG. 6 may further include basic components structurally similar to a typical decoder-only transformer. For example, a positional embedding may be added to each embedding to reflect location information, and each operation block may include a multi-head attention structure composed of multiple heads, an Add & Norm structure that performs residual connection and normalization, and a feed-forward neural network (FFN) structure that includes a nonlinear activation function. Although these detailed configurations are omitted in the drawing, they may be equally applied to the block transformer architecture (180) as to the existing transformer architecture.

[0155] The block transformer architecture (180) according to various embodiments of the present disclosure may include a structure that hierarchically performs block-level global attention and block-internal local attention for multiple input token embeddings to efficiently generate predicted tokens in a self-regressive manner. To this end, the block transformer architecture (180) may include the following three components.

[0156] Referring to FIG. 6, the block transformer architecture (180) may include an embedding unit (Embedder) that merges a plurality of input token embeddings (A to P) into a predetermined number of units (N) to generate a plurality of block embeddings (B1, B2, B3, B4), a block decoder that performs a global attention operation on the plurality of block embeddings (B1, B2, B3, B4) to generate a plurality of context embeddings (C1, C2, C3, C4), and a token decoder that outputs a predicted token embedding by applying the plurality of context embeddings (C1, C2, C3, C4) to a plurality of input token embedding groups (G1, G2, G3, G4) corresponding to the plurality of block embeddings (B1, B2, B3, B4) and then performing a local attention operation within each group.

[0157] Here, the embedding may correspond to a block embedding generation layer (10), the block decoder may correspond to a context embedding generation layer (20), and the token decoder may correspond to a prediction token embedding generation layer (30).

[0158] The block transformer architecture (180) can be trained in advance to receive multiple input token embeddings (A to P) generated through tokenization and vectorization of input data in the form of a sequence, and generate multiple predicted token embeddings (E to T) in an autoregressive manner.

[0159] Below, the process of learning a pre-learned block transformer architecture (180) to generate predicted data in a self-regressive manner through a predetermined operation based on input data is described.

[0160] For learning of a block transformer architecture (180), multiple input token embeddings (A to P) corresponding to a given sequence data may be given as input, and multiple correct token embeddings (E to T) located one block behind the corresponding input may be given as correct answers (ground truth).

[0161] Here, the fact that multiple correct token embeddings (E~T) are positioned one block behind multiple input token embeddings (A~P) means that the multiple input token embeddings (A~P) are composed of token embeddings that are later in time by a predetermined number of units (N) that are merged to create a single block embedding described later. These multiple correct token embeddings (E~T) can be used as output targets of a block transformer architecture (180) according to a block-level autoregressive learning structure.

[0162] The embedding device can generate multiple block embeddings (B1, B2, B3, B4) by sequentially merging multiple input token embeddings (A to P) into a predetermined number of units (N).

[0163] For example, the embedding device can generate a first block embedding (B1) by merging multiple input token embeddings (A to D) included in a first input token group (G1), generate a second block embedding (B2) by merging multiple input token embeddings (E to H) included in a second input token group (G2), generate a third block embedding (B3) by merging multiple input token embeddings (I to L) included in a third input token group (G3), and generate a fourth block embedding (B4) by merging multiple input token embeddings (M to P) included in a fourth input token group (G4).

[0164] Here, for example, a plurality of input token embeddings (A to L) belonging to the first to third input token groups (G1, G2, G3) may be prompt data input by the user, and a plurality of tokens (M to T) following the third input token group (G3) among the plurality of correct answer token embeddings (E to T) given as correct answer data may be prediction data for the prompt data.

[0165] For example, for multiple input token embeddings (A to P), the embedding can generate a block embedding by simply concatenating or integrating a predetermined number of consecutive units (N) of input token embeddings through means such as mean pooling. The block embeddings generated in this way can be used as input for block-level global attention operations.

[0166] The block decoder can perform self-attention operations on multiple block embeddings (B1, B2, B3, B4) generated by the embedding, thereby generating multiple context embeddings (C1, C2, C3, C4) corresponding to each block embedding.

[0167] Here, the block decoder can effectively grasp the global semantic relationship between multiple input token groups (G1, G2, G3, G4) by modeling the entire input sequence in block units, and each context embedding can compress and include global context information necessary for predicting tokens of the subsequent group.

[0168] In addition, multiple context embeddings (C1, C2, C3, C4) generated in this manner are input to a token decoder, and serve to provide global context information when performing local attention operations on multiple input token embedding groups (G1, G2, G3, G4), thereby enabling precise language generation while maintaining efficient inference performance.

[0169] The token decoder can be trained to apply information about multiple context embeddings (C1, C2, C3, C4) to multiple input token embedding groups (G1, G2, G3, G4) corresponding to multiple block embeddings (B1, B2, B3, B4), and then perform local attention operations within each group to output multiple correct token embeddings (E~T). Here, the local attention operations can be performed as masked self-attention operations on the input embeddings.

[0170] In this case, the token decoder takes as input information about one of the plurality of context embeddings (C1, C2, C3, C4) a context embedding, and sequentially generates a plurality of input token embeddings corresponding to subsequent block embeddings of the block embedding for the context embedding, and can generate subsequent token embeddings by referring to the generated previous token embeddings.

[0171] For example, a token decoder can sequentially generate multiple input token embeddings (E, F, G, H) corresponding to a second block embedding (B2), which is a subsequent block embedding of a first block embedding (B1) for the first context embedding (C1), by inputting information about a first context embedding (C1).

[0172] In this case, the token decoder can sequentially generate multiple input token embeddings (E, F, G, H) and generate subsequent token embeddings by referring to the generated previous token embeddings.

[0173] For example, the token decoder can be trained to first generate an E token embedding based on information about a first context embedding (C1), then generate an F token embedding based on information about the first context embedding (C1) and the E token embedding, then generate a G token embedding based on information about the first context embedding (C1), the E token embedding, and the F token embedding, and finally generate an H token embedding based on information about the first context embedding (C1), the E token embedding, the F token embedding, and the G token embedding.

[0174] In this way, the token decoder can be trained in a self-regressive manner by sequentially generating token embeddings from I to L, referring to the second context embedding (C2) as the global context, and utilizing previously generated token embeddings (at least one of I, J, K) as input at each point.

[0175] In addition, the token decoder can be trained in a self-regressive manner by sequentially generating M token embeddings to P token embeddings while referring to the third context embedding (C3) as a global context, and utilizing previously generated token embeddings (at least one of M, N, O) as input at each point in time.

[0176] Furthermore, the token decoder can be trained in a self-regressive manner by sequentially generating token embeddings from Q to T while referring to the fourth context embedding (C4) as a global context, and utilizing previously generated token embeddings (at least one of Q, R, and S) at each point in time as input.

[0177] Meanwhile, the token decoder applies information about multiple context embeddings (C1, C2, C3, C4) to multiple input token embedding groups (G1, G2, G3, G4) corresponding to multiple block embeddings (B1, B2, B3, B4), and generates multiple context-embedded embeddings (P) through linear transformation for multiple context embeddings (C1, C2, C3, C4). 1 , P 2 , P 3 , P 4 ) can be used as input for local attention operations within each group.

[0178] For example, referring to FIG. 7, a first context embedding (C1) is linearly transformed by a predetermined linear layer to produce multiple first context embeddings (P1). 1 , P2 1 , … P k 1 ) can be created.

[0179] In this case, the token decoder embeds multiple first context embeddings (P1 1 , P2 1 , … P k 1 ) to generate E token embeddings, and the first context embedding embedding (P1 1 , P2 1 , … P k 1 ) and generate the F token embedding based on the E token embedding, and the first context embedding embedding (P1 1 , P2 1 , … P k 1 ), E token embedding, and F token embedding, and generate G token embedding, and first context embedding embedding (P1 1 , P2 1 , … P k 1 ), the H token embedding can be generated based on the E token embedding, the F token embedding, and the G token embedding.

[0180] Similarly, the second context embedding (C2) is linearly transformed by a predetermined linear layer to form multiple second context embedded embeddings (P1). 2 , P2 2 , … P k 2 ) is generated, and the third context embedding (C3) is linearly transformed by a predetermined linear layer to form multiple third context embeddings (P1 3 , P2 3 , … P k 3 ) is generated, and the fourth context embedding (C4) is linearly transformed by a predetermined linear layer to generate multiple fourth context embeddings (P1 4 , P2 4 , … P k 4 ) can be created.

[0181] In this way, multiple context embeddings (C1, C2, C3, C4) can each be projected into at least one prefix token embedding, and at least one prefix token embedding for each context embedding can be used as an input value for a local attention operation.

[0182] FIG. 8 is a graph showing the learning perplexity of a language model according to a method of reflecting multiple context embeddings (C1, C2, C3, C4) in an attention operation in a token decoder in a block transformer architecture (180) according to one embodiment.

[0183] Referring to Figure 8, it shows how three approaches, namely, cross-attention, summation, and providing prefix token embeddings generated by linearly transforming multiple context embeddings (C1, C2, C3, C4) as input to the token decoder, affect the language model performance.

[0184] For example, the cross-attention method along the first curve (L1) exhibits relatively poor performance, recording the highest perplexity compared to other methods throughout the entire training period. This suggests that the cross-attention method, which references context embeddings as external keys / values ​​at every decoding step, consumes significant computational resources, but may be inefficient in terms of learning stability and expressiveness.

[0185] Additionally, the summation method according to the second curve (L2) also shows slightly higher perplexity than the prefix token embedding method according to the third to sixth curves (L1 to L6).

[0186] Meanwhile, according to the third to sixth curves (L1 to L6), we can see how the number of prefix token embeddings affects the language model performance.

[0187] For example, the third curve (L3) corresponds to a case where the number of prefix token embeddings is 1, the fourth curve (L4) corresponds to a case where the number of prefix token embeddings is 2, the fifth curve (L5) corresponds to a case where the number of prefix token embeddings is 4, and the sixth curve (L6) corresponds to a case where the number of prefix token embeddings is 6.

[0188] As the number of prefix token embeddings increases (1 → 2 → 4 → 6), the perplexity of the language model steadily decreases, and it is shown that as the number of prefix token embeddings increases, the context information of the input sequence can be reflected more richly in the local attention calculation process.

[0189] Figure 9 is a graph comparing the change in perplexity of a language model according to the length of the block embedding and the distribution ratio of learning parameters between the block decoder and the token decoder.

[0190] Referring to Figure 9, we can see how the perplexity of the language model changes depending on the parameter ratio of the block decoder and the token decoder (5:1, 2:1, 1:1, 1:2, 1:5) for different block embedding lengths (2, 4, 8).

[0191] Here, the length of the block embedding refers to the number of input token embeddings merged to create the block embedding. For example, a block embedding length of 4 corresponds to merging four input token embeddings to create a single block embedding.

[0192] For example, when the length of the block embedding is 4, the performance is best when the learning parameters are evenly allocated 1:1 to the block decoder and token decoder, recording the lowest perplexity.

[0193] Accordingly, when the length of the block embedding is 4, the balance between the representational ability of the block decoder that summarizes global context information and the local processing ability of the token decoder that finely restores individual tokens based on that information can be most appropriately maintained.

[0194] That is, in the block transformer architecture (180), setting the length of the block embedding to 4 can enable the language model to exhibit higher performance with greater stability. If the length of the block embedding is too short, the global context representation of the context embedding becomes insufficient (more learning parameter resources are required for the block decoder), and conversely, if the length is too long, local restoration within the block becomes difficult (more learning parameter resources are required for the token decoder), and it may be difficult to set a balance point for distributing learning parameters. Therefore, setting the length of the block embedding to 4 can provide a structural advantage that effectively balances the trade-off between the computational resources of the language model and information representation.

[0195]

[0196] - Prediction method based on block transformer architecture (S100)

[0197] Hereinafter, a goal-oriented conversation method (S100) capable of providing a more accurate response to a user conversation input is described in detail by extracting and learning features of various types of conversation data sets, generating a conversation graph that models a predetermined conditional relationship for the conversation data set based on the extracted features, and selecting an optimal conversation behavior among multiple conversation behaviors sampled by a pre-trained conversation model for a user conversation input based on the conversation graph and providing the optimal conversation behavior as a response.

[0198] Hereinafter, a computing system (1000) according to one embodiment of the present invention will be described in detail a goal-oriented conversation method (S100) in which a conversation model provides an appropriate response to a user conversation input and performs a task requested by a user based on a conversation graph that models at least one conditional relationship for a conversation data set.

[0199] Fig. 10 is a flowchart of a block transformer architecture-based prediction method (S100) according to one embodiment. Fig. 11 is a flowchart of a step (S107) of generating a subsequent token embedding based on any one of the context embeddings included in the block transformer architecture-based prediction method of Fig. 10.

[0200] Referring to FIG. 10, in one embodiment, a block transformer architecture-based prediction method (S100) may include a step (S101) of processing input data (ID) in a sequence form to generate a plurality of input token embeddings (A to P), a step (S103) of sequentially merging the plurality of input token embeddings (A to P) into a predetermined number of units (N) to generate a plurality of block embeddings (B1, B2, B3, B4), a step (S105) of performing a self-attention operation on the plurality of block embeddings (B1, B2, B3, B4) to generate a plurality of context embeddings (C1, C2, C3, C4) corresponding to each block embedding, and a step (S107) of generating a subsequent prediction token embedding for the plurality of input token embeddings (A to P) based on the plurality of context embeddings (C1, C2, C3, C4).

[0201] In one embodiment, the method (S100) may be performed by a processor (131) included in a server computing system (130). However, the present invention is not limited thereto, and at least a part of the method (S100) may be performed by a processor (111) of a user computing device (110), and another part may be performed by a processor (131) included in the server computing system (130).

[0202] Hereinafter, the processor (131) of the server computing system (130) is described as performing the method (S100).

[0203] In step (S101), the processor (131) of the server computing system (130) can generate multiple input token embeddings (A to P).

[0204] For example, input data (ID) in the form of a predetermined sequence can be tokenized by the processor (131).

[0205] A plurality of input tokens generated as a result of tokenization for input data (ID) can be converted into a plurality of input token embeddings (A to P) by the processor (131).

[0206] For example, each token is mapped to a vector space of fixed dimensions, and the corresponding embedding can be generated by a pre-trained embedding table or a convolution-based or linear projection-based embedding layer.

[0207] In step (S103), the processor (131) of the server computing system (130) can generate a plurality of block embeddings (B1, B2, B3, B4).

[0208] The processor (131) of the server computing system (130) can sequentially merge multiple input token embeddings (A to P) into a predetermined number of units (N) to generate multiple block embeddings (B1, B2, B3, B4). This merging process is a preprocessing step for dividing the entire sequence into block units of a certain size to increase the efficiency of subsequent attention operations.

[0209] For example, token embeddings of a given number of consecutive units (N) can be combined into a single block embedding by simple concatenation or mean pooling. These block embeddings can then be used as input for block-level global attention operations.

[0210] In step (S105), the processor (131) of the server computing system (130) can generate multiple context embeddings (C1, C2, C3, C4).

[0211] A processor (131) of a server computing system (130) can perform a global attention operation on a plurality of block embeddings (B1, B2, B3, B4) to generate a plurality of context embeddings (C1, C2, C3, C4) including global context information reflecting the relationship between the plurality of block embeddings (B1, B2, B3, B4).

[0212] For example, a processor (131) of a server computing system (130) can learn what semantic relationship each block has with its previous blocks by performing a masked self-attention operation between multiple block embeddings (B1, B2, B3, B4), and can output a context embedding corresponding to each block from the result of this operation.

[0213] In step (S107), the processor (131) of the server computing system (130) can generate subsequent prediction token embeddings for the plurality of input token embeddings (A to P) based on the plurality of context embeddings (C1, C2, C3, C4).

[0214] In detail, the processor (131) of the server computing system (130) can sequentially output predicted token embeddings by performing local attention operations on a plurality of input token embedding groups (G1, G2, G3, G4) corresponding to a plurality of block embeddings (B1, B2, B3, B4) based on a plurality of context embeddings (C1, C2, C3, C4).

[0215] For example, the processor (131) of the server computing system (130) may input information of one of a plurality of context embeddings (C1, C2, C3, C4), apply it to an input token group corresponding to a subsequent block embedding of a block embedding for the context embedding, and then perform a local attention operation within the token group to generate a predicted token embedding in an autoregressive manner.

[0216] Meanwhile, in step (S107), the processor (131) of the server computing system (130) can sequentially generate a plurality of input token embeddings corresponding to subsequent block embeddings of block embeddings for each of the plurality of context embeddings (C1, C2, C3, C4).

[0217] For example, a processor (131) of a server computing system (130) may sequentially generate a plurality of input token embeddings (E, F, G, H) corresponding to a second block embedding (B2), which is a subsequent block embedding of a first block embedding (B1) for the first context embedding (C1), based on information about the first context embedding (C1).

[0218] Thereafter, the processor (131) of the server computing system (130) can sequentially generate a plurality of input token embeddings (I, J, K, L) corresponding to a third block embedding (B3), which is a subsequent block embedding of the second block embedding (B2) for the second context embedding (C2), based on information about the second context embedding (C2).

[0219] Similarly, the processor (131) of the server computing system (130) can sequentially generate a plurality of input token embeddings (M, N, O, P) corresponding to the fourth block embedding (B4) based on information about the third context embedding (C3), and can sequentially generate a plurality of input token embeddings (Q, R, S, T) corresponding to subsequent block embeddings to be generated following the fourth block embedding (B4) based on information about the fourth context embedding (C4).

[0220] In this case, the processor (131) of the server computing system (130) linearly transforms multiple context embeddings (C1, C2, C3, C4) to create multiple context embeddings (P 1 , P 2 , P 3 , P 4 ) can be generated. Multiple context-embedding embeddings (P 1 , P 2 , P 3 , P 4 ) can be provided as input to local attention operations within the corresponding token group.

[0221] For example, referring to FIG. 11, step (S107) may include a step (S1071) of generating at least one context embedding based on one context embedding, and a step (S1073) of generating a subsequent token embedding in a self-regressive manner based on a self-attention operation on at least one context embedding and a sequentially generated previous token embedding.

[0222] For example, in step (S1071), a first context embedding (C1) among multiple context embeddings (C1, C2, C3, C4) is linearly transformed to form multiple first context embeddings (P1). 1 , P2 1 , … P k 1 ) can be created.

[0223] Additionally, in step (S1073), the converted multiple first context insert embeddings (P1 1 , P2 1 , … P k 1 ) is generated based on the E token embedding, and the first context embedding (P1 1 , P2 1 , … P k 1 ) and the F token embedding is generated based on the E token embedding, and the first context embedding (P1 1 , P2 1 , … P k 1 ), E token embedding, and F token embedding are generated based on the G token embedding, and the first context embedding (P1 1 , P2 1 , … P k 1 ), H token embedding can be generated based on E token embedding, F token embedding, and G token embedding.

[0224] Similarly, the second context embedding (C2) is linearly transformed to form multiple second context embedded embeddings (P12 , P2 2 , … P k 2 ) are generated, and multiple second context insertion embeddings (P1 2 , P2 2 , … P k 2 ) based on the autoregressive method, I token embedding, J token embedding, K token embedding, and L token embedding can be sequentially generated.

[0225] Additionally, the third context embedding (C3) is linearly transformed to create multiple third context embeddings (P1). 3 , P2 3 , … P k 3 ) are generated, and multiple third context insertion embeddings (P1 3 , P2 3 , … P k 3 ) based on which M token embedding, N token embedding, O token embedding, and P token embedding can be sequentially generated in an autoregressive manner.

[0226] Furthermore, the fourth context embedding (C4) is linearly transformed to create multiple fourth context embeddings (P1 4 , P2 4 , … P k 4 ) are generated, and multiple fourth context insertion embeddings (P1 4 , P2 4 , … P k 4 ) can be sequentially generated as Q token embedding, R token embedding, S token embedding, and T token embedding using an autoregressive method.

[0227] In addition, the method (S100) may further include a step of generating an additional subsequent block embedding after the last sequence by merging a plurality of input token embeddings generated based on a context embedding corresponding to the last sequence among a plurality of context embeddings (C1, C2, C3, C4), and a step of generating a plurality of context embeddings corresponding to each block embedding by performing a self-attention operation again on the plurality of block embeddings and the additional subsequent block embeddings.

[0228] For example, referring to FIG. 6, the processor (131) of the server computing system (130) can generate a plurality of input token embeddings (Q, R, S, T) based on the fourth context embedding (C4), which is the last sequence among the plurality of context embeddings (C1, C2, C3, C4), and can merge them to generate an additional subsequent block embedding (not shown).

[0229] Additionally, the processor (131) of the server computing system (130) can perform self-attention operations again on the previously generated plurality of block embeddings (B1, B2, B3, B4) and additional subsequent block embeddings to generate new plurality of context embeddings that reflect the extended input sequence.

[0230] In this way, in step (S107), the plurality of input token embeddings (E~T) generated for the plurality of context embeddings (C1, C2, C3, C4) by the processor (131) of the server computing system (130) may be referred to as a plurality of output token embeddings from an input / output perspective. Based on these plurality of output token embeddings, prediction data (PD) for the input data (ID) may be generated.

[0231] For example, if the input data (ID) is text, each of the multiple output token embeddings can be linearly transformed to a dimension equal to the vocabulary size within the language model, thereby generating multiple logit embeddings. A softmax function can be applied to these multiple logit embeddings to generate a probability distribution for each word in the vocabulary, and based on this, text-based prediction data (PD) can be generated.

[0232] The embodiments of the present invention described above may be implemented in the form of program commands that can be executed by various computer components and recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. Hardware devices may be changed into one or more software modules to perform processing according to the present invention, and vice versa.

[0233] The specific implementations described in the present invention are exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components illustrated in the drawings are merely representative of functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. In addition, unless specifically mentioned as “essential,” “important,” etc., a component may not be absolutely necessary for the application of the present invention.

[0234] Although the detailed description of the present invention has been described with reference to preferred embodiments of the present invention, it will be understood by those skilled in the art or having ordinary knowledge in the art that various modifications and changes can be made to the present invention without departing from the spirit and technical scope of the present invention as set forth in the claims below. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

[0235] The block transformer architecture-based prediction method and system of the present invention has industrial applicability in that it can significantly reduce computational complexity and memory I / O by generating prediction data for sequence-type input based on a plurality of context embeddings generated by performing a global attention operation on a plurality of block embeddings in which sequence-type input token embeddings are merged into predetermined units.

Claims

1. A transformer architecture-based prediction method performed by a computing device including a memory and a processor, A step of processing input data in sequence form to generate multiple input token embeddings; A step of generating a plurality of block embeddings by sequentially merging the plurality of input token embeddings into a predetermined number of units (N); A step of generating a plurality of context embeddings corresponding to each block embedding by performing a self-attention operation on the plurality of block embeddings; and A block transformer architecture-based prediction method comprising: generating a subsequent prediction token embedding for the plurality of input token embeddings based on the plurality of context embeddings; 2. In paragraph 1, The step of generating a subsequent predicted token embedding for the above multiple input token embeddings comprises: A block transformer architecture-based prediction method, comprising: a step of sequentially generating a plurality of input token embeddings corresponding to subsequent block embeddings of a block embedding for the context embedding, by inputting information about one of the plurality of context embeddings, and generating the subsequent token embeddings by referring to the generated previous token embeddings; 3. In paragraph 2, A step of generating an additional subsequent block embedding after the last sequence by merging the plurality of input token embeddings generated based on the context embedding corresponding to the last sequence among the plurality of context embeddings; and A block transformer architecture-based prediction method, further comprising: a step of generating a plurality of context embeddings corresponding to each block embedding by performing a self-attention operation again on the plurality of block embeddings and the additional subsequent block embeddings; 4. In paragraph 2, In the step of generating the above subsequent token embedding, A prediction method based on a block transformer architecture, which sequentially generates a plurality of input token embeddings corresponding to subsequent block embeddings of a block embedding for each of the plurality of context embeddings.

5. In paragraph 1, A prediction method based on a block transformer architecture, wherein each of the plurality of block embeddings is generated by concatenating a plurality of input token embeddings of a predetermined number of units (N) arranged in order.

6. In paragraph 1, A prediction method based on a block transformer architecture, wherein the above-mentioned number of units (N) is 4.

7. In paragraph 2, The step of generating the subsequent token embedding based on any one of the above context embeddings comprises: generating at least one context-embedded embedding based on any one of the above context embeddings; and A block transformer architecture-based prediction method, comprising: generating the subsequent token embedding in a self-regressive manner based on a self-attention operation on the at least one context-embedded embedding and the sequentially generated previous token embedding; 8. In paragraph 7, In the step of generating at least one context-embedding embedding, A block transformer architecture-based prediction method, wherein at least one context-embedded embedding is generated through a linear transformation of any one of the context embeddings.

9. In paragraph 8, In the step of generating at least one context-embedding embedding, A prediction method based on a block transformer architecture, wherein multiple context-embedded embeddings are generated through linear transformation of any one of the above context embeddings.

10. In paragraph 7, A prediction method based on a block transformer architecture, wherein learning parameters are evenly distributed to a block decoder that performs self-attention operations on the plurality of block embeddings and a token decoder that performs self-attention operations on the at least one context-embedded embedding and the sequentially generated previous token embeddings.

11. In paragraph 1, The step of generating the above multiple input token embeddings is: A step of processing input data in sequence form to generate a plurality of input tokens; and A block transformer architecture-based prediction method, comprising: a step of generating a plurality of input token embeddings based on the plurality of input tokens; 12. At least one memory; and At least one processor configured to perform a transformer architecture-based prediction method by reading at least one instruction stored in the memory; At least one processor, Process input data in sequence form to generate multiple input token embeddings, Generate multiple block embeddings by sequentially merging the above multiple input token embeddings into a predetermined number of units (N), Performing a self-attention operation on the above multiple block embeddings to generate multiple context embeddings corresponding to each block embedding, A prediction system based on a block transformer architecture that generates subsequent prediction token embeddings for the plurality of input token embeddings based on the plurality of context embeddings.

13. In paragraph 12, wherein said at least one processor generates a subsequent predicted token embedding for said plurality of input token embeddings, A prediction system based on a block transformer architecture, which sequentially generates a plurality of input token embeddings corresponding to subsequent block embeddings of a block embedding for the context embedding by inputting information about one of the plurality of context embeddings, and generates subsequent token embeddings by referencing the generated previous token embeddings.

14. In paragraph 12, A plurality of neurons comprising an array including at least one register, at least one programmable logic, and at least one input interface; a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; and at least one routing network that controls data flow between the plurality of neurons; A block transformer architecture-based prediction system, wherein each of the plurality of neurons further includes a field programmable gate array (FPGA) implementation for a predetermined artificial neural network that is connected to at least one other neuron through the routing network to set a transmission path of the weight.

15. In paragraph 12, A plurality of neurons organized in an array including at least one register, at least one microprocessor, and at least one input; and a plurality of synaptic circuits storing synaptic weights that adjust connection strengths between the plurality of neurons; A block transformer architecture-based prediction system, wherein each of the plurality of neurons further includes an application specific integrated circuit (ASIC) for a given artificial neural network, wherein the application specific integrated circuit is connected to at least one other neuron through one of the plurality of synaptic circuits.

Images