QUERY PROCESSING WITH ARTIFICIAL INTELLIGENCE THROUGH MEMORY DEVICE WITH NEAR-MEMORY PROCESSING

Abstract

Systems, methods, and apparatus are provided for processing artificial intelligence queries by a near-memory storage device. In one or more examples, the systems, apparatus, and methods comprise receiving data at a first near-memory storage (PNM) device and processing first values ​​from the data with transposed query values ​​from the data at the first PNM storage device. In one or more examples, the systems, apparatus, and methods comprise determining a probability distribution of a result of the processing at the first PNM storage device and generating an activation value based on the probability distribution at the first PNM storage device, wherein the activation value indicates a correlation between text units in a query associated with the data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 608,821, filed December 11, 2023, which is incorporated herein by reference for all purposes. TECHNICAL FIELD

[0002] The disclosure relates generally to storage systems and, more particularly, to artificial intelligence query processing by a storage device with near-memory processing. BACKGROUND

[0003] This Background section is intended to provide context only, and disclosure of a concept in this section does not constitute an admission that the concept is prior art.

[0004] Artificial intelligence is a branch of computer science that mimics human intelligence and can learn and adapt from data. With AI search, the platform learns from user data to automatically generate the most accurate and relevant search results. AI search can involve query processing, information retrieval, and ranking. Query processing involves analyzing a user's query to understand its intent, scope, and constraints. AI search uses machine learning and algorithms to crawl and categorize large amounts of data. AI search engines use natural language processing (NLP) and other AI-based algorithms to understand search queries and deliver accurate results.

[0005] The information disclosed in this Background section is provided solely to facilitate understanding of the background of the disclosure and may therefore contain information that is not prior art. SUMMARY

[0006] In various embodiments, systems, methods, and apparatus for processing artificial intelligence queries by a memory device with near-memory processing are described herein.

[0007] In some aspects, the techniques described herein relate to a method for query processing, the method comprising: receiving data at a first near-memory processing (PNM) storage device; processing first values ​​from the data with transposed query values ​​from the data at the first PNM storage device; determining a probability distribution of a result of the processing at the first PNM storage device; and generating an activation value based on the probability distribution at the first PNM storage device, the activation value indicating a correlation between text units in a query associated with the data.

[0008] In some aspects, the techniques described herein relate to a method wherein the first PNM storage device includes a memory that stores key-value information from the data. In some cases, the data includes attention data.

[0009] In some aspects, the techniques described herein relate to a method wherein the memory and a processor of the first PNM memory device comprise an integrated circuit based on the memory being stacked on the processor and the memory being communicatively coupled to the processor.

[0010] In some aspects, the techniques described herein relate to a method further comprising: receiving at least a second activation value from a second PNM memory device via a die-to-die (D2D) communication interface; and forming a unified activation value based at least in part on a combination of the activation value of the first PNM memory device and the second activation value of the second PNM memory device, wherein the D2D communication interface enables communication between the first PNM memory device and the second PNM memory device.

[0011] In some aspects, the techniques described herein relate to a method further comprising receiving a trigger from the processing device prior to receiving the data, the trigger including at least one of the following information: a user identification or a layer number information associated with the data.

[0012] In some aspects, the techniques described herein relate to a method in which the data is a portion of the multi-head attention data distributed among the multiple PNM storage devices.

[0013] In some aspects, the techniques described herein relate to a method wherein the data includes a portion of query attention data from a first attention layer, a portion of key attention data from a second attention layer, and a portion of value attention data from a third attention layer.

[0014] In some aspects, the techniques described herein relate to a method in which the data is based on an iteration of activation values ​​generated based on partial outputs generated by the plurality of PNM memory devices that are reduced to a unified output.

[0015] In some aspects, the techniques described herein relate to a method wherein the plurality of PNM storage devices comprise an array of solid-state drives.

[0016] In some aspects, the techniques described herein relate to a method wherein the first PNM memory device is a system-on-chip die that includes a solid-state drive and at least one processor.

[0017] In some aspects, the techniques described herein relate to a method in which the first PNM memory device is communicatively coupled to the processing device via a high bandwidth expansion bus.

[0018] In some aspects, the techniques described herein relate to a method where the processing device includes at least one graphics processing unit (GPU) communicatively coupled to a high bandwidth memory.

[0019] In some aspects, the techniques described herein relate to a method where the data includes attention data, the first values ​​include key values, the second values ​​include query values, and the key-value data includes a key-value matrix.

[0020] In some aspects, the techniques described herein relate to a query processing system, the query processing system comprising: a processing device communicatively coupled to a first near-memory (PNM) storage device, the processing device to transfer data to the first PNM storage device; and the plurality of near-memory (PNM) storage devices, the first PNM storage device configured to: process first values ​​from the data with transposed query values ​​from the data; determine a probability distribution of a result of the processing; and generate an activation value based on the probability distribution, the activation value indicating a correlation between text units in a query associated with the data.

[0021] In some aspects, the techniques described herein relate to a query processing system, wherein the first PNM storage device includes a memory that stores key-value data from the data.

[0022] In some aspects, the techniques described herein relate to a query processing system in which the memory and a processor of the first PNM memory device comprise an integrated circuit based on the memory being stacked on the processor and the memory being communicatively coupled to the processor.

[0023] In some aspects, the techniques described herein relate to a query processing system, wherein the first PNM memory device is configured to: receive at least a second activation value from a second PNM memory device via a die-to-die (D2D) communication interface; and form a unified activation value based at least in part on a combination of the activation value of the first PNM memory device and the second activation value of the second PNM memory device, wherein the D2D communication interface enables the first PNM memory device and the second PNM memory device to communicate.

[0024] In some aspects, the techniques described herein relate to a query processing system, wherein the first PNM storage device is configured to: receive a trigger from the processing device prior to receiving the data, wherein the trigger includes at least one of a user identifier or layer number information associated with the data.

[0025] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium storing code, the code including instructions executable by a processor of a first PNM storage device to: receive data; process, at the first PNM storage device, first values ​​from the data with transposed query values ​​from the data; determine, at the first PNM storage device, a probability distribution of a result of the processing; and generate, at the first PNM storage device, an activation value based on the probability distribution, wherein the activation value indicates a correlation between text units in a query associated with the data.

[0026] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium, wherein: the first PNM storage device includes a memory that stores key-value data from the data, and the memory and a processor of the first PNM storage device include an integrated circuit based on the memory being stacked on the processor and the memory being communicatively coupled to the processor.

[0027] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium, wherein the code includes further instructions executable by the processor to cause the first PNM memory device to: receive, via a die-to-die (D2D) communication interface, at least a second activation value from the second PNM memory device; and form a unified activation value based at least in part on a combination of the activation value of the first PNM memory device and the second activation value of the second PNM memory device, wherein the D2D communication interface enables the first PNM memory device and the second PNM memory device to communicate.

[0028] A computer-readable medium is disclosed. The computer-readable medium can store instructions that, when executed by a computer, cause the computer to perform substantially the same or similar operations as described herein. Similarly, non-transitory computer-readable media, apparatus, and systems for performing substantially the same or similar operations as described herein are further disclosed.

[0029] The techniques described here offer numerous advantages and benefits. For example, the AI ​​inference delegation techniques provide scalable memory bandwidth and storage capacity to accommodate larger query lengths and a larger number of concurrent users. The AI ​​inference delegation techniques reduce system costs by reducing the use of expensive graphics processing unit (GPU) systems. Furthermore, the AI ​​inference delegation techniques support relatively long queries with sufficient memory for advanced generative, pre-trained transformers. The AI ​​inference delegation techniques enable the multiplication of a query key-value (QKV) with a PNM-SSD (near-memory processing SSD). Furthermore, the AI ​​inference delegation techniques enable the performance of softmax and layered normalization operations in a PNM-SSD.AI inference delegation techniques avoid backing up inactive KV data, or key-value data, to external storage. Furthermore, AI delegation techniques minimize communication overhead between a main compute node (e.g., a GPU system) and a memory-bandwidth-intensive system (e.g., PNM SSD). Accordingly, AI inference delegation techniques reduce system costs and provide scalable memory bandwidth and capacity for advanced generative, pre-trained transformers by allowing the simple addition of more PNM SSDs to a given system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other aspects of the present technology will be better understood when the present application is read in light of the following figures, in which like numerals indicate similar or identical elements. Furthermore, the drawings contained herein are for illustrative purposes only; other embodiments that may not be expressly shown are not excluded from the scope of this disclosure.

[0031] These and other features and advantages of the present disclosure will be appreciated and understood by reference to the specification, claims, and accompanying drawings, in which: Fig. 1 shows an example system in accordance with one or more of the implementations described herein. Fig. 2 shows details of the system of Fig. 1 according to one or more implementations described herein. Fig. Figure 3 shows an example system in accordance with one or more of the implementations described herein. Fig. Figure 4 shows an example system in accordance with one or more of the implementations described herein. Fig. Figure 5 shows an example system in accordance with one or more of the implementations described herein. Fig. Figure 6 shows a flowchart illustrating an example method in connection with the systems described herein in accordance with the example implementations described herein. Fig. Figure 7 shows a flowchart illustrating an example method in connection with the systems described herein in accordance with the example implementations described herein.

[0032] While the present techniques are susceptible to various modifications and alternative forms, specific embodiments thereof are exemplified in the drawings and described herein. The drawings may not be to scale. However, it should be understood that the drawings and their detailed description are not intended to limit the present techniques to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present techniques as defined by the appended claims. DETAILED DESCRIPTION OF DIFFERENT EMBODIMENTS

[0033] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

[0034] Various embodiments of the present disclosure are described in more detail below with reference to the accompanying drawings, in which some, but not all, embodiments are illustrated. Indeed, the disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will conform to applicable legal requirements. The term "or" is used herein in both the alternative and conjunctive senses unless otherwise indicated. The terms "illustrative" and "example" are used as examples without indication of quality. Like numbers refer to like elements throughout. Arrows in the figures represent bidirectional data flow and / or bidirectional data flow capabilities.The terms “path”, “way” and “route” are used interchangeably here.

[0035] Embodiments of the present disclosure may be implemented in various ways, including as computer program products comprising articles of manufacture. A computer program product may comprise a non-transitory, computer-readable storage medium that stores applications, programs, program components, scripts, source code, program code, object code, bytecode, compiled code, interpreted code, machine code, executable instructions, and / or the like (also referred to herein as executable instructions, instructions to execute, computer program products, program code, and / or similar terms, which are used interchangeably herein). Such non-transitory computer-readable storage media include any computer-readable media (including volatile and non-transitory media).

[0036] In one embodiment, a non-transitory, computer-readable storage medium may include a floppy disk, a flexible disk, a hard disk, a solid-state storage (SSS) (e.g., a solid-state drive (SSD)), a solid-state card (SSC), a solid-state module (SSM), an enterprise flash drive, magnetic tape, or any other non-transitory magnetic medium, and / or the like. A non-transitory, computer-readable storage medium may include a punched card, a paper tape, an optical marker sheet (or other physical medium with hole patterns or other optically recognizable indicia), a compact disc read-only memory (CD-ROM), a rewritable compact disc (CD-RW), a digital versatile disc (DVD), a Blu-ray disc (BD), another non-transitory optical medium, and / or the like.Such a non-volatile computer-readable storage medium may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory (e.g., serial, NAND, NOR, and / or similar), multimedia memory cards (MMC), secure digital (SD) memory cards, SmartMedia cards, CompactFlash (CF) cards, memory sticks, and / or similar.Additionally, a non-transitory computer-readable storage medium may include conductive bridging random access memory (CBRAM), phase change random access memory (PRAM), ferroelectric random access memory (FeRAM), non-volatile random access memory (NVRAM), magnetoresistive random access memory (MRAM), resistive random access memory (RRAM), silicon oxide nitride oxide silicon (SONOS) memory, floating junction gate random access memory (FJG RAM), millipede memory, racetrack memory, and / or the like.

[0037] In one embodiment, a volatile computer-readable storage medium may include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), fast page mode dynamic random access memory (FPM DRAM), extended data output dynamic random access memory (EDO DRAM), synchronous dynamic random access memory (SDRAM), synchronous double data rate synchronous dynamic random access memory (DDR SDRAM), type two synchronous double data rate synchronous dynamic random access memory (DDR2 SDRAM), third generation double data rate synchronous dynamic random access memory (DDR3 SDRAM), Rambus dynamic random access memory (RDRAM), twin transistor RAM (TTRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), Rambus inline memory device (RIMM), dual inline memory device (DIMM), single inline memory device (SIMM),Video Random Access Memory (VRAM), cache memory (including various levels), flash memory, register memory, and / or the like. It will be appreciated that where embodiments are described using a computer-readable storage medium, other types of computer-readable storage media may be used instead of, or in addition to, the computer-readable storage media described above.

[0038] As should be appreciated, various embodiments of the present disclosure may be implemented as methods, apparatus, systems, computing devices, computing units, and / or the like. As such, embodiments of the present disclosure may take the form of an apparatus, system, computing device, computing unit, and / or the like that execute instructions stored on a computer-readable storage medium to perform particular steps or operations. Thus, embodiments of the present disclosure may take the form of a hardware-only implementation, a computer program product-only implementation, and / or an embodiment comprising a combination of computer program products and hardware to perform particular steps or operations.

[0039] Embodiments of the present disclosure are described below with reference to block diagrams and flowchart illustrations. It should therefore be understood that each block of the block diagrams and flowchart illustrations may be implemented in the form of a computer program product, a complete hardware implementation, a combination of hardware and computer program products, and / or devices, systems, computing devices, computing units, and / or the like that execute instructions, operations, steps, and similar words used interchangeably (e.g., executable instructions, instructions to execute, program code, and / or the like) on a computer-readable storage medium for execution. For example, the retrieval, loading, and execution of the code may occur sequentially, such that one instruction is retrieved, loaded, and executed at a time.In some embodiments, the fetching, loading, and / or execution may occur in parallel, such that multiple instructions are fetched, loaded, and / or executed together. Thus, such embodiments may result in specifically configured machines that perform the steps or operations specified in the block diagrams and flowcharts. Accordingly, the block diagrams and flowchart illustrations support various combinations of embodiments for performing the specified instructions, operations, or steps.

[0040] The following description is intended to enable one skilled in the art to make and use the subject matter disclosed herein and to incorporate it into the context of particular applications. While the following description refers to specific examples, other and further examples may be developed without departing from the general scope of application.

[0041] Various modifications and a variety of uses in different applications will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to a variety of embodiments. Therefore, the subject matter disclosed herein is not intended to be limited to the embodiments presented, but is intended to have the widest possible scope consistent with the principles and novel features disclosed herein.

[0042] In this description, numerous specific details are set forth in order to provide a more thorough understanding of the subject matter disclosed herein. However, it will be apparent to one skilled in the art that the subject matter disclosed herein may be practiced without limitation to these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail in order not to obscure the subject matter disclosed herein.

[0043] All features disclosed in this description (e.g., in the appended claims, the abstract, and the drawings) may be replaced by alternative features serving the same, equivalent, or similar purpose, unless expressly stated otherwise. Unless expressly stated otherwise, each disclosed feature is only one example of a general series of equivalent or similar features.

[0044] Various features are described herein with reference to the figures. It should be noted that the figures are provided only to facilitate the description of the features. The various features described are not intended to be an exhaustive description of the subject matter disclosed herein or to limit the scope of the subject matter disclosed herein. Furthermore, an illustrated example may not have all of the illustrated aspects or advantages. An aspect or advantage described in connection with a particular example is not necessarily limited to that example and may be realized in any other example, even if not illustrated or not expressly described.

[0045] Furthermore, any element in a claim that is not expressly recited as a "means" of performing a particular function or a "step" of performing a particular function shall not be construed as a "means" or "step" feature within the meaning of 35 USC Section 112, Paragraph 6. In particular, the use of "step of" or "act of" in the present claims shall not be construed as an invocation of the provisions of 35 USC 112, Paragraph 6.

[0046] Please note that the terms left, right, front, back, top, bottom, forward, backward, clockwise, and counterclockwise are used for convenience only and are not intended to imply any specific fixed direction. Instead, the terms are used to indicate relative positions and / or directions between different parts of an object.

[0047] Data processing may include data buffering, alignment of incoming data from multiple communication channels, forward error correction ("FEC"), and / or other features. For example, the data may first be received by an analog front-end (AFE), which prepares the incoming data for digital processing. The digital portion (e.g., DSPs) of the transceivers or transmitters may provide skew management, equalization, reflection cancellation, and / or other features. The method described here can provide many benefits, including power and cost savings.

[0048] In addition, the terms "system," "component," "module," "interface," "model," or similar generally refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. A component can be, for example, but not limited to, a process running on a processor, a processor, an object, an executable file, a thread of execution, a program, and / or a computer. To illustrate, both an application running on a control unit and the control unit can be a component. One or more components can be located in a process and / or thread of execution, and a component can be localized on one computer and / or distributed across two or more computers.

[0049] Unless explicitly stated otherwise, any numerical value and range can be interpreted as approximate, as if the word "about" or "approximately" preceded the value or range. Signals and corresponding nodes or ports may be referred to by the same name and are interchangeable for the purposes of this document.

[0050] Although the embodiments have been described in terms of circuit functions, the embodiments of the subject matter disclosed herein are not limited. Possible implementations may be included in a single integrated circuit, a multi-chip module, a single card, a system-on-chip, or a multi-card circuit package. As will be apparent to one of ordinary skill in the art, the various embodiments may also be implemented as part of a larger system. Such embodiments may be employed, for example, in conjunction with a digital signal processor, a microcontroller, a field-programmable gate array, an application-specific integrated circuit, or a general-purpose computer.

[0051] As will be apparent to one skilled in the art, various functions of circuit elements may also be implemented as processing blocks in a software program. Such software may be employed, for example, in a digital signal processor, a microcontroller, or a general-purpose computer. Such software may be embodied in the form of program code in tangible media such as magnetic recording media, optical recording media, solid-state memories, floppy disks, CD-ROMs, hard disks, or other non-transitory, machine-readable storage media, such that when the program code is loaded into and executed by a machine such as a computer, the machine becomes a device capable of performing the subject matter disclosed herein.When implemented in a general-purpose processor, the program code segments are combined with the processor to create a unique device that functions analogously to specific logic circuits. Described embodiments may also manifest in the form of a bit stream or other sequence of signal values ​​transmitted electrically or optically over a medium, or in the form of stored magnetic field changes in a magnetic recording medium, etc., generated using a method and / or apparatus described herein.

[0052] Artificial intelligence (AI) encompasses the concept of creating intelligent machines that can perceive, reason, act, and adapt. Machine learning (ML) can be a subset of AI that supports the development of AI-driven applications. Deep learning can be a subset of machine learning that uses artificial neural networks to mimic the learning process of the human brain. Deep learning algorithms can use large amounts of data and complex algorithms to train a model. Neural networks can form the basis of deep learning algorithms. In machine learning, AI inference can involve the process of using a trained model to make predictions. In some cases, AI training can be the first step in a two-part machine learning process.Inference can be faster than training because inference doesn't involve the model adjusting its parameters based on new data. Inference also consumes less computational power than training clusters. AI can involve delegation of AI interference. AI inference delegation techniques offer scalable memory bandwidth and storage capacity to increase query lengths and the number of concurrent users.

[0053] AI search may include query processing, retrieval, and ranking. In some cases, the systems and methods described here may include AI query processing through near-memory processing. AI search may process large amounts of data and queries in real time, anticipate user needs based on past search patterns, quickly deliver accurate and relevant results, automatically refine over time, learn from user data to automatically generate the most accurate and relevant search experiences, and so on. AI search systems may process various types of input, such as natural language queries, voice commands, images, contextual information, and the like.

[0054] The systems and methods described here may include AI processes based on neural processing units (NPUs). An NPU can be a specialized processor that executes machine learning algorithms. NPUs can also be referred to as AI accelerators or intelligent processing units (IPUs). NPUs enhance the inference performance of neural networks. NPUs can be configured to operate similarly to the human brain. NPUs can consist of neurons and synapses that send and receive signals to and from each other. NPUs can use a data-driven parallel computing architecture to process large amounts of multimedia data such as images and videos. NPUs can be used to offload certain workloads, allowing dedicated hardware to focus on more specialized tasks.

[0055] In some examples, the systems and methods may include an attention network. An attention network may include a machine learning technique that identifies the strongest correlations between words in a sentence. An attention network may do this by learning patterns from a training corpus. Attention models may evaluate inputs to identify the most important components and assign them a weight. For example, when translating a sentence, an attention model may select the most important words and assign them a higher weight. Attention mechanisms may be additive or dot product-based. With additive attention, a feed-forward neural network may be used to calculate the compatibility between the query and the key vectors. Dot product attention may use a dot product to measure their similarity. Attention mechanisms may also use self-attention or dot product attention.Self-attention. Self-attention can be a mechanism used in machine learning, particularly in natural language processing (NLP) and computer vision tasks. Self-attention can enable the model to recognize and weigh the importance of different parts of the input sequence and how the different parts relate to each other (e.g., relevance between different parts of the input sequence or tokens). In some examples, the systems and methods of the present application can incorporate attention networks to perform the AI ​​inference delegation techniques described herein.

[0056] In some cases, the systems and methods may incorporate machine learning (ML)-based attention. ML-based attention can be a mechanism that mimics cognitive attention. An attention network can compute "soft" weights for each word, or more precisely, for its embedding, in the context window. An attention network can compute either in parallel (as in transformers) or sequentially (as in recurrent neural networks). Soft weights can change during each runtime, unlike hard weights, which are (pre-)trained and fine-tuned and remain frozen thereafter. An attention network can be designed to detect the highest correlations between the words of a sentence, assuming that the attention network has learned these patterns through training.Such correlations can be captured in the neural weights by backpropagation either through self-supervised pre-training or supervised fine-tuning.

[0057] In machine learning, the query, key, and value (QKV) can be used in attention networks to model relationships between words. In some examples, the systems and methods described here enable QKV vector multiplication through a solid-state drive (SSD) with near-memory processing (PNM). Attention networks can assign weights to the words in a sentence, giving more weight to relevant words. The assigned weights can help preserve the context of the sentence and improve the accuracy of predictions. QKV enables an attention network to focus on the most important parts of the input and produce relevant and coherent output. The attention operation can be thought of as a retrieval process.For example, when searching for a video online, the query is the text in the search bar, the keys are the video title and description, and the values ​​are the videos that match the query. With AI transformers, the query is the information being sought, the key is the context or reference, and the value is the content being sought. The query and the key can be multiplied together to obtain attention scores, which are then used to calculate the weighted sum of the scores.

[0058] In some examples, the systems and methods described here may include multi-head attention (MHA). MHA may be a type of attention mechanism that uses multiple attention layers to process information from an input sequence. MHA may allow a neural network to control how information is shuffled between parts of an input sequence. This can lead to richer representations and better performance on machine learning tasks. In some cases, MHA includes one or more modules configured for attention mechanisms that iterate through an attention mechanism multiple times in parallel. In some cases, the independent attention outputs are concatenated and linearly transformed to the expected dimension. MHAs allow for different levels of attention to be paid to parts of the sequence (e.g., longer-term dependencies versus shorter-term dependencies).Multi-head attention can comprise multiple parallel attention levels (heads). Each head can have different linear transformations on the queries, keys, values, and outputs. For example, one head may focus on the relationship between people, while another head considers the context of the sentence, and so on. Multi-head attention combines the knowledge of the same attention pool across different representation subspaces of queries, keys, and values. To compute multiple heads of multi-head attention in parallel, appropriate tensor manipulation can be used.

[0059] In some examples, the systems and methods described herein enable softmax and layered normalization operations to be performed in a PNM SSD. The softmax function may include a function that converts a vector of K real values ​​into a vector of K real values ​​that sum to 1 (e.g., into a probability distribution of K possible outcomes). The input values ​​can be positive, negative, 0, or greater than 1, and the softmax function converts them to values ​​between 0 and 1 so that they can be interpreted as probabilities. The softmax function can provide probability values ​​that range from 1 to 0 (e.g., 0, 0.0925, 0.1, 0.95, 1, etc.), whereas the max function can only provide a binary output of 1 for max and 0 otherwise, with no possible values ​​in between.The softmax activation function can be a mathematical function that converts a vector of real numbers into a probability distribution. The softmax function can be used as an activation function in the output layer of neural network models that predict a multinomial probability distribution. The softmax activation function can be used for multiclass classification problems where class membership is based on more than two class labels. The softmax function can raise each element to a power, making it positive, and then normalizes it by dividing by the sum of all exponentiated values. The output of a softmax function can be a vector of probabilities for each possible outcome. The probabilities in the vector v sum to one for all possible outcomes or classes. The softmax function can be an extension of the sigmoid function.Sigmoid can be used for binary classification methods where there are two classes.

[0060] The systems and methods described herein may include an activation function. The activation function may be a function in a neural network used to determine the output of a neuron. In some cases, an activation value may be based on a probability distribution. The activation value may indicate relevance between text units in a query associated with attention data. An AI system may determine whether a neuron is activated or not based on a weighted sum of inputs. In some cases, the systems and methods may include layer normalization. Layer normalization may be a technique for normalizing the activations of a layer of a neural network. Layer normalization may be performed by normalizing the activations for each individual sample in a batch by subtracting the mean and dividing by the standard deviation.Examples of activation functions include the tanh function, the sigmoid function, the exponential linear unit (ELU), the linear activation function, the maxout function, and the binary step activation function. The tanh function can be a nonlinear activation function that can be used between the layers of a neural network. The tanh function has a similar shape to the sigmoid function, but its range is -1 to 1. The sigmoid activation function can be used in neural networks. The sigmoid function can be applied to the output of each neuron, allowing the network to introduce nonlinearity into the model. The ELU activation function can be used for nonlinear estimation. The output layer of the ELU activation function can contain a single node and provides an estimated level from the given inputs.The linear activation function is a simple, straight-line activation function where the function is directly proportional to the weighted sum of the neurons or the input. The maxout function allows for the nonlinearity to be applied as a dot product between the weights of a neural network and the data. The binary step function can be used as an activation function when building a binary classifier.

[0061] In some examples, the systems and methods may include large language models (LLMs). LLMs may include some type of AI algorithm (e.g., a deep learning algorithm) that can understand, summarize, generate, and predict new content. LLMs may use statistical models to analyze large amounts of data and detect patterns and connections between words and sentences. LLMs may be based on machine learning, specifically a type of neural network called a transformer model. In some cases, the systems and methods implement transformer models with LLMs.

[0062] In some cases, an LLM may include a feedforward layer (FFN), consisting of multiple fully connected layers that transform the input embeddings. In this way, these layers enable the model to capture higher-level abstractions (e.g., to understand the user's intent with the text input). The systems and methods described here may implement Gaussian Error Linear Unit (GELU) as the activation function. The GELU activation function may be xΦ(x), where Φ(x) is the standard cumulative Gaussian distribution function. In dropout regularization, the inputs of a neuron are stochastically multiplied by 0, causing them to become randomly inactive. ReLU activation deterministically multiplies the inputs by 0 or 1, depending on the value of the input. GELU combines dropout regularization and Rectified Linear Unit (ReLU) by multiplying inputs by a value between 0 and 1.The value of this zero-one mask is determined stochastically, but can also depend on the value of the input.

[0063] In some cases, the systems and methods may implement an LLM that includes a reduction layer. For example, a map-reduce-document chain first applies an LLM chain to each document individually (the map step) and treats the chain output as a new document. The LLM then passes all new documents to a separate chain for combining documents to obtain a single output (the reduce step).

[0064] In some cases, the systems and methods described here may be based on recurrent neural networks (RNNs). RNNs can be a type of artificial neural network designed to process sequential data. RNNs can detect sequential features in data and use patterns to predict the next likely scenario. RNNs may have feedback connections that allow them to retain information from previous time steps. This allows RNNs to capture temporal dependencies. RNNs may consist of a series of repeating neural network cells connected in a chain-like structure. The output of one cell is passed as input to the next cell.

[0065] In some cases, the systems and methods for delegating AI inference described here are based on tokens. Tokens can be basic units of text or code that AI models (e.g., LLMs) use to process and generate language. Tokenization is the division of input / output text into smaller units for LLM AI processing. The vocabulary size is the number of tokens each model uses and varies from model to model. Examples of tokens include characters, words, subwords, other text segments or codes, punctuation, parts of words, clauses, phrases, etc. The tokenization method or scheme used determines the type of token. In some examples, the phrase [I love you] might have five tokens: [I], [Love], [You], [ ], and [.]. The tokens can be converted into an embedding, which the LLM model then processes to understand the text.Each LLM has a maximum limit on the number of tokens it can process.

[0066] In some examples, the systems and methods may rely on key-value caching (KV caching). KV caching may include a method for caching data to improve application response times. KV caching involves caching frequently accessed data in a key-value store. KV caching reduces database queries, complex calculations, and conserves computational resources by reusing previously computed key-value pairs instead of recalculating them for each generated token. Key and value state are used to compute dot product attention.A decoding phase can generate a single token at each time step, but each token depends on the key and value tensors of all previous tokens (including the input token's KV tensors computed during prefill and all new KV tensors computed up to the current time step). At each token generation step, the query vector of a single current token can be multiplied by the key vectors of all previous tokens in the sequence to generate attention scores, and the scores are further multiplied by the value vectors of all previous tokens. Thus, instead of recomputing the key and value vectors for all previous characters at each character generation step, KV caching can rely on performing only one incremental computation for a current character and reusing previously computed key / value vectors from the KV cache.The KV vector of the current token can also be appended to the KV cache for the next token generation step. The AI ​​inference delegation systems and methods described here can support relatively long queries with sufficient memory for advanced generative, pre-trained transformers.

[0067] In some cases, the systems and methods implement near-memory computing (NMC). NMC can be a system architecture that moves computational capacity into or near memory (e.g., random-access memory (RAM)). This enables memory-centric computing and addresses the bandwidth bottleneck between the central processing unit (CPU) and memory. NMC can include processing near-memory (PNM) and processing in-memory (PIM). PIM can improve performance and energy efficiency by moving some of the data computation tasks from the CPU to memory. PIM allows computations and processing to be performed within the memory of a computer, server, or similar device. PNM can integrate memory and logic chips (e.g., processing units) into an integrated circuit package (e.g.,System-on-Chip (SoC) can be integrated, which reduces data movement between the CPU and memory by using memory for data computation, resulting in improved system performance and greater energy efficiency. PNM can enable computational functions (e.g., AI processing) to be performed closer to memory, reducing the bottleneck that occurs between the CPU and data transfer in memory. PNM can be used in caching, multithreading, embedded random-access memory, and AI processing to mitigate the problem of CPU memory bottlenecks.

[0068] A solid-state drive (SSD) may contain a non-volatile storage medium that stores persistent data on solid-state flash. A PNM SSD is an SSD that includes PNM. Thus, a PNM SSD may contain memory and logic chips in an integrated circuit package (e.g., SoC, an SoC SSD with memory, logic chips, processing units, microcontrollers, solid-state flash, etc.). A single-memory system (SoC) may be an integrated circuit (IC) that contains the components of a computer system. An SoC may include at least a central processing unit (CPU), a field-programmable gate array (FPGA), a graphics processing unit (GPU), random access memory (RAM), storage (e.g., SSD), network interfaces, input / output (I / O) ports, peripheral interfaces, secondary storage devices, and / or I / O drivers, etc. A system-in-a-package (SiP) can be a method of bundling multiple ICs (e.g., multiple SoCs) and passive components into a single package.The SiP can assume the functions of an entire system. The AI ​​inference delegation systems and techniques described here reduce system costs by using PNM SSDs and reducing the use of expensive GPU (graphical processing unit) systems.

[0069] In some cases, the systems and methods described herein may implement a RAM channel. The RAM channel may refer to an aspect of the memory architecture called a memory channel. The memory channel may be the number of communication channels available between a RAM module and the memory controller. The memory controller may be the digital circuitry that manages the flow of data to and from the RAM module. The memory rank may refer to the number of sets of memory chips the module contains. The memory channels of the memory allow the memory controller to access the different ranks of the memory and provide a data conduit between the memory and the CPU. The multi-channel memory architecture may increase the number of channels the memory controller can utilize, resulting in an increase in the data transfer rate.A dynamic random access memory (DRAM) channel may include a controller interface that can communicate with one or more ranks. A DRAM channel may be a common group of address / data lines that function together. The systems and methods may implement through-silicon via (TSV). TSV may include a chip packaging technology that vertically interconnects integrated chip dies (e.g., processors, DRAM chip dies, etc.). TSV can be an alternative to wire bonding and flip chips for creating 3D packages and 3D integrated circuits. Hybrid copper bonding is a process in which chips are connected in packages with copper-to-copper interconnects. Hybrid copper bonding can be used for packages with relatively small pitches (e.g., 10 µm and below).

[0070] Some AI inference systems (e.g., generative AI, generative pre-trained transformers, LLMs, etc.) may encounter performance constraints related to memory bandwidth and capacity. As query length and / or the number of concurrent users increase, AI inference systems may encounter memory bandwidth and / or capacity limitations. In some cases, QKV data processing may be a limiting factor in scalability in terms of concurrent users and / or query length.

[0071] In Transformer Attention systems, processing user queries (e.g., QKV processing) may involve relatively high memory bandwidth and / or storage capacity constraints due to the potential for relatively long queries. However, in some cases, QKV processing may have relatively low computational constraints (e.g., with a low computational load).

[0072] Some approaches use a main processing unit (e.g., GPU, CPU of a high-performance computing system, etc.) to compute weighting operations and QKV operations (e.g., all processes of a transformer attention system). As generatively pre-trained transformers evolve, query lengths steadily increase (e.g., from a query length of 2K to a query length of 120K). Storage space becomes increasingly scarce to handle longer queries and larger numbers of concurrent users. High-bandwidth memory (HBM) can be used, but HBM capacity is limited, and the host's cloud storage may be too slow to handle the increased memory bandwidth and storage capacity constraints.

[0073] Because weight operations are computationally rather than memory-intensive (e.g., they require relatively high processing levels), weight operations are better performed by a main processing unit (e.g., CPU, GPU) of a compute node. Because QKV operations are memory-intensive rather than computationally intensive (e.g., they require relatively high memory bandwidths and / or storage capacities), QKV operations are better performed by a PNM (near-memory processing) storage device.

[0074] Fig. 1 shows an example system 100 in accordance with one or more of the implementations described herein. In Fig. 1 shows a device 105, which may be referred to as a host, system, or server. While in Fig. 1, the device 105 is depicted as a tower computer, embodiments of the disclosure may extend to any form factor or device type. For example, the machine 105 may be a rack server, a blade server, a desktop computer, a tower computer, a mini-tower computer, a desktop server, a laptop computer, a notebook computer, a tablet computer, etc.

[0075] The machine 105 may include a processor 110, a memory 115, and a storage device 120. The processor 110 may be any type of processor. It should be noted that the processor 110, along with the other components discussed below, is depicted external to the machine for simplicity. Embodiments of the disclosure may include these components within the machine. While Fig. 1 shows a single processor 110, the machine 105 may include any number of processors, each of which may be single-core or multi-core, each of which may implement a RISC (Reduced Instruction Set Computer) or a CISC (Complex Instruction Set Computer) architecture (among other possibilities), and which may be mixed in any desired combination.

[0076] The processor 110 may be connected to the memory 115. The memory 115 may be any type of memory, such as flash memory, dynamic random access memory (DRAM), static random access memory (SRAM), persistent random access memory, ferroelectric random access memory (FRAM), or non-volatile random access memory (NVRAM), such as magnetoresistive random access memory (MRAM), phase-change memory (PCM), or resistive random access memory (ReRAM). The memory 115 may include volatile and / or non-volatile memory. The memory 115 may have any desired form factor: e.g., single in-line memory module (SIMM), dual in-line memory module (DIMM), non-volatile DIMM (NVDIMM), etc. The memory 115 may be any combination of different memory types and may be managed by the memory controller 125.Memory 115 may store data that may be described as "short-term," meaning data that is not expected to be stored for extended periods of time. Examples of short-term data include temporary files, data used locally by applications (which may have been copied from other storage locations), and the like.

[0077] The processor 110 and the memory 115 may support an operating system under which various applications may run. These applications may issue queries (which may be referred to as commands) to read or write data from the memory 115 or the storage device 120. If the storage device 120 is used to support applications that read or write data via some type of file system, access to the storage device 120 may be via the device driver 130. While Fig. 1 shows a storage device 120, there may be any number (one or more) of storage devices in the machine 105. The storage device 120 may support any desired protocol or protocols, for example, the Non-Volatile Memory Express (NVMe) protocol, a Serial Attached Small Computer System Interface (SCSI) (SAS) protocol, or a Serial AT Attachment (SATA) protocol. The storage device 120 may have any interface, such as a Peripheral Component Interconnect Express (PCIe®) interface or a Compute Express Link (CXL) interface. The storage device 120 may have any form factor, such as a U.2 form factor, a U.3 form factor, an M.2 form factor, an Enterprise and Data Center Standard Form Factor (EDSFF) (including all of its variants, such as E1 short, E1 long, and E3), or an Add-In Card (AIC).

[0078] While in Fig. 1, the term "storage device" is used, embodiments of the disclosure may include any storage device formats that may benefit from the use of computational storage units, e.g., hard disk drives, solid-state drives (SSDs), or persistent memory devices such as PCM, ReRAM, or MRAM. Any reference to "storage device" "SSD" below should be understood to include such other embodiments of the disclosure and other types of storage devices. In some cases, the term "storage device" may include the storage device 120 and the memory 115. The machine 105 may include a power supply 135. The power supply 135 may provide power to the machine 105 and its components.

[0079] The machine 105 may include a transmitter 145 and a receiver 150. The transmitter 145 and the receiver 150, respectively, may be used to send or receive data (e.g., query processing data). In some cases, the transmitter 145 and / or the receiver 150 may be used to communicate with the memory 115 and / or the storage device 120. The transmitter 145 may include a write circuit 160 that may be used to write data to the memory, e.g., a register, in the memory 115 and / or the storage device 120. Similarly, the receiver 150 may include a read circuit 165 that may be used to read data from the memory, e.g., a register, in the memory 115 and / or the storage device 120.

[0080] In one or more examples, machine 105 may be implemented with any type of device. Machine 105 may be configured as (e.g., host of) one or more of a server, such as a compute server, a storage server, a storage node, a network server, a supercomputer, a data center system, and / or the like, or any combination thereof. Additionally or alternatively, machine 105 may be configured as (e.g., host of) one or more computers, such as a workstation, a personal computer, a tablet, a smartphone, and / or the like, or any combination thereof. Machine 105 may be implemented with any type of device that can be configured as a device, e.g.,an accelerator device, a memory device, a network device, a memory expansion and / or buffer device, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), and / or the like, or any combination thereof.

[0081] Any communication between devices including the machine 105 (e.g., host, computational storage device, and / or any intermediate device) may occur via an interface that may be implemented using any type of wired and / or wireless communication medium, interface, protocol, and / or the like, including PCIe, NVMe, Ethernet, NVMe-oF, Compute Express Link (CXL), and / or a coherent protocol such as CXL.mem, CXL.cache, CXL.IO and / or the like, Gen-Z, Open Coherent Accelerator Processor Interface (OpenCAPI), Cache Coherent Interconnect for Accelerators (CCIX), Advanced eXtensible Interface (AXI) and / or the like, or a combination thereof, Transmission Control Protocol / Internet Protocol (TCP / IP), FibreChannel, InfiniBand, Serial AT Attachment (SATA), Small Computer Systems Interface (SCSI), Serial Attached SCSI (SAS), iWARP, any generation of wireless networks including 2G, 3G, 4G, 5G, and / or the like, any generation of Wi-Fi, Bluetooth, Near Field Communication (NFC), and / or the like, or a combination thereof. In some embodiments, the communication interfaces may comprise a communication fabric having one or more links, buses, switches, hubs, nodes, routers, translators, repeaters, and / or the like.In some embodiments, system 100 may include one or more additional devices with one or more additional communication interfaces.

[0082] Any functionality described herein, including host functionality, device functionality, query processing controller 140 functionality, and / or the like, may be implemented with hardware, software, firmware, or any combination thereof, for example, combinational logic, sequential logic, timers, counters, registers, state machines, volatile memory such as dynamic random access memory (DRAM) and / or static random access memory (SRAM), non-volatile memory including flash memory, persistent memory such as cross-grid non-volatile memory, bulk resistance change memory, phase change memory (PCM), and / or the like, and / or any combination thereof, complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), CPUs,including complex instruction set computer (CISC) processors such as x86 processors and / or reduced instruction set computer (RISC) processors such as RISC-V and / or ARM processors, graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), and / or the like, executing instructions stored in any memory. In some embodiments, one or more components of query processing controller 140 may be implemented as a system-on-chip (SoC).

[0083] In some examples, query processing controller 140 may include any combination of logic (e.g., logic circuitry), hardware (e.g., processing unit, memory, storage), software, firmware, and the like. In some cases, query processing controller 140 may perform one or more functions in conjunction with processor 110. In some cases, at least a portion of query processing controller 140 may be implemented in or by processor 110 and / or memory 115. The one or more logic circuits of query processing controller 140 may include any combination of multiplexers, registers, logic gates, arithmetic logic units (ALUs), cache, computer memory, microprocessors, processing units (CPUs, GPUs, NPUs, and / or TPUs), FPGAs, ASICs, etc.that enable the query processing controller 140 to provide artificial intelligence query processing through near-memory processing.

[0084] In one or more examples, query processing controller 140 may enable artificial intelligence query processing in conjunction with near-memory processing. For example, query processing controller 140 may provide the AI ​​inference delegation techniques described herein. In one or more examples, query processing controller 140 receives attention data from a GPU processing device, processes key values ​​from the attention data with translated query values ​​from the attention data, determines a probability distribution of a result of the processing, and generates an activation value (e.g., activation values, activation vector) based on the probability distribution, wherein the activation value indicates relevance between text units in a query associated with the attention data.Accordingly, the query processing controller 140 provides scalable memory bandwidth and storage capacity to accommodate larger query lengths and / or a larger number of concurrent users, thereby reducing system costs by reducing the use of relatively expensive compute-intensive graphical processing unit (GPU) systems, avoiding the backup of inactive KV data to external storage, and minimizing communication overhead between a main compute node (e.g., GPU system) and a system with relatively high memory bandwidth (e.g., PNM SSD array).

[0085] Fig. 2 shows details of machine 105 from Fig. 1, according to the examples described here. In Fig. 2, the machine 105 typically includes one or more processors 110, which may include a memory controller 125 and clocks 205, which may be used to coordinate the operations of the components of the machine. The processors 110 may be coupled to memories 115, which may include, for example, random access memory (RAM), read-only memory (ROM), or other state-preserving media. The processors 110 may be connected to storage devices 120 and to a network connection 210, which may be, for example, an Ethernet connection or a wireless connection. The processors 110 may be connected to buses 215 to which user interfaces 220 and input / output (I / O) interfaces are connected, which may be managed by, among other things, I / O engines 225.As shown, the processors 110 may be coupled to the query processing controller 230, which is an example of the query processing controller 140 of FIG. Fig. 1. Additionally or alternatively, the processors 110 may be connected to buses 215 to which the query processing controller 230 may be connected.

[0086] Fig. 3 shows an example system 300 in accordance with one or more of the implementations described herein. In one or more examples, system 300 illustrates an example system for processing artificial intelligence queries over a storage device with near-memory processing. In the illustrated example, system 300 may include a PNM SSD SoC 305. As shown, PNM SSD SoC 305 may include a key-value (KV) cache 310, at least one NPU 315, a memory channel 320, a communication interface 325 (e.g., PCIe communication interface), and at least one interconnect 330 (e.g., one or more die-to-die (D2D), chip-to-chip (C2C), chiplet-to-chiplet, SoC interconnects, etc.). As illustrated, PNM-SSD SoC 305 may include at least one NPU (e.g., NPU 315a, NPU 315b, NPU 315c through NPU 315N), where N is a positive integer. Memory channel 320 may include a persistent memory channel (e.g.,Flash memory, SSD, etc.) and / or a DRAM channel (e.g., RAM, SRAM, etc.), etc. In some examples, one or more components of system 300 (e.g., PNM SSD SoC 305, KV cache 310, at least one NPU, memory channel 320, etc.) may include a query processing controller 140 of FIG. Fig. 1 and / or a query processing controller 230 of Fig. 2 and / or cooperate with them. In some cases, the KV cache 310, the NPU 315a through NPU 315N, and the memory channel 320 may be configured to perform artificial intelligence query processing (e.g., QKV processing) based on the near-memory processing capabilities of the system 300.

[0087] Weighting operations (e.g., weight matrix operations) are more limited by compute power (e.g., measured in teraflops) than by memory bandwidth. However, QKV operations are more limited by memory bandwidth and / or storage capacity than by compute power. Accordingly, the techniques described herein delegate weight processing to one or more processing units (e.g., CPUs, GPUs of main compute nodes, etc.) and delegate QKV processing to one or more PNM storage devices (e.g., PNM SSD, PNM SSD SoC, PNM SSD SoC array) such as PNM SSD SoC 305. Accordingly, PNM SSD SoC 305 may be configured to perform QKV processing for an LLM query, while one processing unit (e.g., a compute-intensive GPU system) may process the LLM query's weight operations.

[0088] In one or more examples, PNM-SSD SoC 305 may be configured to support relatively long queries with sufficient memory for advanced AI inference systems, generative AI, generative pre-trained transformers, LLMs, etc. The PNM-SSD SoC of system 300 is configured to perform query processing (e.g., QKV vector multiplication) for one or more queries, while a compute-intensive system (e.g., a GPU system with high-bandwidth memory) performs weighting processing for the one or more queries. The PNM-SSD SoC 305 may be configured to perform softmax functions and layered normalization operations (e.g., offloading a main processing unit, a processing unit of a compute-intensive hardware system).

[0089] Fig. 4 shows an example system 400 in accordance with one or more of the implementations described herein. In the illustrated example, system 400 may include a GPU system 405 and a QKV system 410. GPU system 405 may be configured as a compute-intensive GPU-based system for multiplying weight matrices. QKV system 410 may be configured as a scalable QKV processing system. As illustrated, GPU system 405 may include at least one GPU (e.g., GPU 415a through GPU K, where K is a positive integer). In some examples, one or more components of system 400 (e.g., GPU system 405, QKV system 410, one or more components of GPU system 405, and / or one or more components of QKV system 410) may include a query processing controller 140. Fig. 1 and / or a query processing controller 230 Fig. 2 and / or cooperate with them.

[0090] As shown, GPU 415a may include at least one HBM (e.g., HBM 420a, HBM 420b, HBM 420c, HBM 420d, HBM 420e, through HBM 420L, where L is a positive integer). As shown, GPU 415K may include at least one HBM (e.g., HBM 425a, HBM 425b, HBM 425c, HBM 425d, HBM 425e through HBM 425L). As illustrated, each GPU may include L HBM units, where L is a positive integer. In some cases, GPU 415K may have fewer or more HBM units than GPU 415a.

[0091] In the illustrated example, the GPU system 405 is connected to the QKV system 410 via at least one communication interface 480. Communication via the communication interface 480 can be from the GPU system 405 to the QKV system 410 and / or from the QKV system 410 to the GPU system 405. In some cases, the communication interface 480 can implement the NVMe protocol (e.g., 16 GB / s for up to 300 users per instance of the communication interface 480). In some cases, the GPU system 405 can communicate values ​​or vectors of Q, K, and / or V to the QKV system 410 (e.g., vector Q[N], vector K[N], vector V[N], where N for QKV values ​​is a positive integer that depends on the query length and / or the number of users and can be independent of other instances of N used herein as an integer). In some cases, the QKV system 410 may transmit activation values ​​to the GPU system 405 (e.g.,Vector Activation[N], where N is a positive integer for activation values). In this case, N may be independent of other uses or other instances of the variable N (e.g., N number of memory modules differs from N in Activation[N]). With respect to Q[N], K[N], V[N], and Activation[N], N may be the number of concurrent users and / or queries in a batch process. In some cases, N may change (e.g., increase or decrease) with each iteration of query processing. At each iteration, the QKV values ​​received from the QKV system 410 may be based on the attention data from a previous iteration (e.g., updated attention data).

[0092] As shown, QKV system 410 includes at least one PNM SSD SoC (e.g., a PNM SSD array including PNM SSD SoC 430a, PNM SSD SoC 430b, etc.). In some cases, PNM SSD SoC 430a is a single-chip system die that includes a solid-state drive and at least one processor. In some cases, PNM SSD SoC 430b is a single-chip system die that includes a solid-state drive and at least one processor. The number of PNM SSD SoCs in QKV system 410 may be based on a need identified at a particular time (e.g., adding and / or enabling PNM SSD SoCs, pluggable PNM SSD SoCs, hot-swappable PNM SSD SoCs, etc., as needed). In some cases, the number of PNM SSD SoCs in the QKV system 410 may increase as demand (e.g., the number of queries, the number of users) increases over time.

[0093] In the illustrated example, PNM SSD SoC 430a includes KV cache 435 and at least one NPU (e.g., NPU 440a, NPU 440b, NPU 440c, through NPU 440M, where M is a positive integer). As shown, PNM SSD SoC 430a may include a communication interface 445 (e.g., PCIe communication interface) and an interconnect 450 (e.g., D2D, C2C, SoC interconnect, etc.). Communication interface 445 may be an example of communication interface 325. Interconnect 450 may be an example of interconnect 330. As illustrated, the communication interface 445 may enable communication between the GPU system 405, the QKV system 410, the PNM SSD SoC 430a, and / or the PNM SSD SoC 430b.

[0094] In the illustrated example, PNM-SSD SoC 430b includes KV cache 465 and at least one NPU (e.g., NPU 470a, NPU 470b, NPU 470c, through NPU 470M, where M is a positive integer). In some cases, PNM-SSD SoC 430b may have more or fewer NPUs compared to PNM-SSD SoC 430a. In some cases, PNM-SSD SoC 430b may include an interconnect 460a and an interconnect 460b (e.g., D2D, C2C, SoC interconnects). As shown, interconnect 460a may enable PNM-SSD SoC 430b to communicate with PNM-SSD SoC 430a (e.g., D2D, C2C, SoC interconnect communication, etc.). As shown, the interconnect 460b may enable the PNM SSD SoC 430b to communicate with another PNM SSD SoC added to the QKV system 410 (e.g., via D2D, C2C, SoC interconnect communication, etc.). In some examples, the QKV system 410 includes multiple PNM SSD SoC chips (e.g., PNM SSD SoC 430a, PNM SSD SoC 430b, etc.) connected via a D2D interconnect (e.g.,Connection 450, connection 460a, connection 460b, etc.) are clustered in a system in a package (SiP).

[0095] In one or more examples, each PNM SSD SoC may include an array of memory modules (e.g., NAND flash, DRAM, other types of persistent or non-persistent memory). In the illustrated example, PNM SSD SoC 430a may be connected to and / or include at least one memory device (e.g., memory 455a, memory 455b, memory 455d, memory 455N, where N is a positive integer). The memory of PNM SSD SoC 430a (e.g., memory 455a to memory 455N) may be an example of memory channel 320. In the illustrated example, PNM SSD SoC 430b may be connected to and / or include at least one memory device (e.g., memory 475a, memory 475b, memory 475d, to memory 475N, where N is a positive integer).

[0096] In some examples, system 400 delegates the calculation of weight processing of a query (e.g., a batch of queries) to GPU system 405 and QKV processing of the query (e.g., a batch of queries) to QKV system 410. In some cases, QKV system 410 stores key-value matrices (KV matrices associated with QKV processing) via the PNM SSD array, enabling the processing of relatively large query lengths and / or concurrent users associated with increasingly advanced generative pre-trained transformers. Based on QKV system 410, there is no need to back up inactive KV data to external storage, as is the case with some approaches. Instead, the KV data is stored in the KV cache (e.g., KV cache 435, KV cache 465).

[0097] In one or more examples, KV cache 435 and / or KV cache 465 may be an on-die cache memory (e.g., cache memory, SRAM, etc.) configured to buffer or store key-value matrices (KV matrices and / or key-value tables in conjunction with QKV processing). In some cases, KV cache 435 and / or KV cache 465 may be implemented to avoid redundantly loading a key-value table from NAND (e.g., because NAND is slower than cache memories such as KV cache). In some examples, KV cache 435 and / or KV cache 465 may include on-die cache built on a corresponding PNM SSD SoC based on a 3D stacking process (e.g., through-silicon via (TSV), hybrid copper bonding, etc.). In some cases, KV cache 435 and a PNM SSD SoC 430a of the processor (e.g.,At least one of the NPU 440a, NPU 440b, NPU 440c, and NPU 440M) may be formed as a stacked integrated circuit, with the KV cache 435 stacked above the processor and the KV cache communicatively connected to the processor. For example, at least a portion of the KV cache 435 may be stacked on top of the NPU 440a, and the KV cache 435 may be communicatively connected to the NPU 440a through one or more vertical connections between the KV cache 435 and the NPU 440a.

[0098] The memory of the PNM SSD SoC 430b (e.g., memory 475a, memory 475b, memory 475d, through memory 475N) may be an example of a memory channel 320. In some cases, the number of memory devices connected to the PNM SSD SoC 430b (e.g., memory 475a, memory 475b, memory 475d, through memory 475N) may be less than or greater than the number of memory devices connected to the PNM SSD SoC 430a (e.g., memory 455a, memory 455b, memory 455d, through memory 455N). In some examples, the at least one memory device of the PNM SSD SoC 430a may be at least partially integrated into the PNM SSD SoC 430a. In some cases, the at least one memory unit of the PNM SSD SoC 430b may be at least partially integrated into the PNM SSD SoC 430b. In some examples, the at least one memory unit of the PNM SSD SoC 430a may be NAND flash, DRAM memory, or another type of persistent memory and / or non-persistent memory.In some cases, the at least one storage unit of the PNM-SSD-SoC 430b may be NAND flash or another type of persistent and / or non-persistent storage.

[0099] In one or more examples, system 400 illustrates an example of a QKV processing system for processing artificial intelligence queries over a storage device with near-memory processing. As shown, GPU system 405 may be configured to process weighting operations (e.g., weight quantization) associated with a query (e.g., a batch of queries). In some examples, the PNM SSD array of QKV system 410 (e.g., PNM SSD SoC 430a, PNM SSD SoC 430b, etc.) may be configured to process QKV operations (e.g., QKV matrix multiplication) associated with the query (e.g., the batch of queries). Each PNM SSD SoC of QKV system 410 provides both relatively high memory / storage bandwidth and high memory / storage capacity.Accordingly, increased memory / storage bandwidth and storage / storage capacity requirements can be easily met by adding one or more additional PNM SSD SoCs to the QKV System 410 (e.g., pluggable PNM SSDs, hot-swappable PNM SSDs). Accordingly, the QKV System 410 provides a scalable system configured to adapt to increasing query lengths and / or an increasing number of concurrent users at relatively low cost.

[0100] Based on the QKV system 410, multiple PNM SSD chips can be clustered via D2D interconnects. For example, interconnect 450 can enable PNM SSD SoC 430a to connect to PNM SSD SoC 430b via interconnect 460a. Similarly, interconnect 460b can enable PNM SSD SoC 430b to connect to another PNM SSD SoC, creating a cluster of interconnected PNM SSD SoCs for QKV processing. In some cases, the PNM SSD SoCs (e.g., SoC dies) of the QKV system 410 may be clustered in a system-in-package (SiP), enabling a PNM SSD SoC (e.g., PNM SSD SoC 430a) to send and / or receive partial computation results to one or more other PNM SSD SoCs (e.g., PNM SSD SoC 430b) without host communication (e.g., bypassing the GPU system 405 and / or a host).

[0101] Based on the QKV system 410, KV matrices (e.g., from one or more queries, a batch of queries, etc.) may be distributed to one or more PNM SSD SoCs. For example, a first portion of the KV matrices of one or more queries may be distributed to PNM SSD SoC 430a, a second portion of the KV matrices of the one or more queries may be distributed to PNM SSD SoC 430b, a third portion of the KV matrices of the one or more queries may be distributed to another PNM SSD SoC (e.g., connected to PNM SSD SoC 430b), and so on.

[0102] Each PNM-SSD SoC of QKV system 410 may be configured to perform query processing on a portion of the KV matrices (e.g., partial matrix multiplication). A PNM-SSD SoC may be configured to perform one or more reduction operations in coordination with one or more other PNM-SSD SoCs (e.g., based on the partial matrix multiplication performed by the two or more PNM-SSD SoCs). The reduction operations between the two or more PNM-SSDs (e.g., between at least PNM-SSD SoC 430a and PNM-SSD SoC 430b, etc.) may be performed based on the D2D interconnections of the two or more PNM-SSD SoCs.

[0103] To minimize loading latency with respect to the memory of the PNM SSD array (e.g., NAND flash memory), the QKV system 410 may precharge at least one of the PNM SSD SoC 410a and / or the PNM SSD SoC 410b based on a "hint ahead" operation (e.g., issued by a compute-intensive chip, issued by the GPU system 405, issued by a host system, etc.). For example, the GPU system 405 may provide a hint (e.g., a precharge trigger) to the QKV system 410 (e.g., before the QKV system 410 receives QKV data, receives KV matrices, and / or begins QKV matrix multiplication, etc.). In some cases, the note may contain a user ID (e.g., one or more user IDs) and / or shift number information associated with one or more queries and / or users (e.g., user ID and shift number information associated with a group of queries / users).

[0104] The techniques described herein include query processing logic to enable artificial intelligence query processing through near-memory processing. The query processing logic includes any combination of hardware (e.g., at least one memory, at least one processor), logic circuitry, firmware, and / or software to enable the processing of artificial intelligence queries through near-memory processing. The query processing may relate to artificial intelligence inferences (e.g., related to attention networks).

[0105] To increase the length of queries and the number of concurrent users, the described techniques split query processing into two parts instead of using a single system (e.g., an expensive GPU base system). Weighting operations are more compute-intensive than memory-intensive (e.g., weighting operations have relatively high processing requirements). Therefore, the weighting operations are performed by a high-performance compute node (e.g., GPU System 405). QKV operations are more memory-intensive than compute-intensive (e.g., QKV operations have relatively high memory bandwidth and / or storage capacity constraints). Therefore, QKV operations can be delegated to and executed by a system with high memory bandwidth (e.g., QKV System 410, PNM-SSD SoC 430a, PNM-SSD SoC 430b, etc.).

[0106] Delegating weighting and QKV operations provides better performance and scalability for LLM inferencing compared to some approaches. Based on the techniques described here, the communication overhead between a main compute node (e.g., a GPU system) and a memory-bandwidth-intensive system (e.g., a PNM SSD) is minimized.

[0107] In some examples, PNM-SSD SoC 430a receives attention data from GPU system 405 (e.g., via PCIe, communication interface 445). In some cases, PNM-SSD SoC 430a processes key values ​​from the attention data with transposed query values ​​from the attention data and determines a probability distribution of a result of the processing. In some examples, PNM-SSD SoC 430a generates an activation value (e.g., activation values, activation vector, Activation[N]) based on the probability distribution. The activation value may indicate relevance between text units in a query associated with the attention data.

[0108] In one or more examples, PNM-SSD-SoC 430a receives at least a second activation value (e.g., second activation values, second activation vector) from PNM-SSD-SoC 430b via connection 450 and forms a unified activation value (e.g., vector Activation2[N]) based at least in part on a combination of the activation value of PNM-SSD-SoC 430a and the second activation value of PNM-SSD-SoC 430b.

[0109] In one or more examples, the attention data received from the PNM SSD SoC 430a is a portion of multi-head attention data distributed across multiple PNM storage devices of the QKV system 410. In some cases, the attention data includes a portion of query attention data from a query multi-head attention layer, a portion of key attention data from a key multi-head attention layer, and a portion of value attention data from a value multi-head attention layer. In some examples, the attention data is based on an iteration of activation values ​​generated based on previous partial outputs generated by the multiple PNM storage devices and reduced to a unified output.

[0110] Fig. 5 shows an example system 500 in accordance with one or more of the implementations described herein. System 500 illustrates an example system for processing artificial intelligence queries via a storage device with near-memory processing. As shown, Fig. 4 a PNM storage device.

[0111] In the example shown, the system 500 includes a multi-head attention query MHA (MHA(Q)) 505a, a key MHA (MHA(K)) 505b, a value MHA (MHA(V)) 505c, a QKV system 510, and an output MHA (MHA(O)). In some examples, the QKV system 510 is an example of the QKV system 410 of Fig. 4. As shown, the QKV system 510 includes the query processing controller 515. The query processing controller 515 may be an example of the query processing controller 140 of Fig. 1 and / or the query processing controller 230 of Fig. 2. In some cases, the query processing controller 515 may include and / or operate in conjunction with one or more PNM SSD SoCs (e.g., a PNM SSD SoC array, PNM SSD SoC 305, PNM SSD SoC 430a, PNM SSD SoC 430b).

[0112] In one or more examples, an input query is assigned to a particular system (e.g., Token Index[N], where N indicates the number of concurrent users and / or queries in a batch process). The input query may be fed to an embedding position encoding process that filters the input query. In some examples, the query data from a batch process may be partitioned into MHA(K) 505a, MHA(V) 505b, and MHA(Q) 505c. In some cases, the inputs to MHA(Q) 505c include an activation value (e.g., the vector Activation1[N] based on Token Index[N]) with a query weight (Wq1) applied to the activation value (e.g., query weights applied to a set of activation values). In some examples, the inputs to MHA(K) 505a include the activation value with key weights (Wk1) applied to the activation value (e.g.Key weights applied to a set of activation values). In some examples, the inputs to MHA(V) 505b include the activation value based on value weights (Wv1) applied to the activation value (e.g., value weights applied to a group of activation values). In some cases, a computationally intensive system (e.g., GPU system 405) calculates the weight values ​​(e.g., Wq1, Wk1, Wv1).

[0113] As shown, MHA(Q) outputs Q1[N], MHA(K) K1[N], and MHA(V) V1[N] to the QKV system 510 (e.g., a near-memory processing storage device, PNM storage device, an array of PNMs, SSDs, and SoCs). In some examples, the query processing controller 515 performs QKV processing based on the inputs Q1[N], K1[N], and V1[N].

[0114] As shown, the QKV system 510 receives inputs based on a query (e.g., the input vector { K[N], V[N], Q[N]}, where N indicates the number of concurrent users / queries in a batch process). In some cases, N indicates a user ID associated with a query (e.g., a value of N is a user ID for a specific query). In the example shown, queries / responses (e.g., all queries / responses { K[0:QL], V[0:QL]}, where QL represents the query length, e.g., 2048, 4096, or 8192 characters) are stored and processed in a low-level storage device. For example, the vector K1[N] and the vector V1[N] are stored in the QKV system 510 (e.g., in an array of PNM SSD SoCs).

[0115] In some examples, QKV processing may include reading the QKV history (e.g., K[0:QL][N], V[0:QL][N]), matrix multiplication, evaluation by softmax, matrix multiplication, and outputting a calculation result. QKV processing may include a matrix multiplication operation (e.g., K[0:QL][N] x QT[N], K1{1~2048} x Q1[N]). QKV processing may include evaluation by a softmax function (e.g., softmax [N], softmax [0:QL][N]). In some cases, the matrix multiplication operations may include softmax[0:QL][N] x V[0:QL][N], and / or softmax[N] x V1{1~2048}). In some examples, K1{1~2048} and V1{1~2048} represent memory records (e.g., metrics for historical queries). As query length increases, a given memory record grows (e.g., from 1 to 2048 characters, based on a system limit of 2048 characters per query).K1{1~4096} and V1{1~4096} mean the dataset can range from 1 to 4096 characters, and so on. When new metrics are added, the QKV data is multiplied by the historical query metrics (e.g., the results of the previous matrix multiplication). For example, K1[N] can be multiplied by the historical query metrics, resulting in K1{1~2048} (e.g., K1 with 1 to 2048 characters). Similarly, V1[N] can be multiplied by the historical query metrics, resulting in V1{1~2048} (e.g., V1 with 1 to 2048 characters).

[0116] In one or more examples, QKV processing may include outputting a computation result (e.g., Output { Activation2[N]}). In some cases, the query processing controller 515 may generate an output (e.g., the MHA output MHA(O)) based on the QKV processing. In some cases, the process may include one or more reduction functions. In some examples, one or more activation functions may be implemented that result in activation outputs (e.g., Vector Activation1[N], Vector Activation2[N], Vector Activation3[N]). In some examples, Activation1[N] is an activation output from the embedding position encoding process, and Activation2[N] is an output of the QKV system 510 based on at least one iteration of QKV processing of an input query (e.g., Token Index[N]).

[0117] In some examples, QKV processing may identify and weigh the importance of different parts of an input sequence, and the output may indicate how the different parts of the input sequence relate to each other (e.g., relevance between the different parts of the input sequence or tokens). In some cases, the computation result may be based on previous iterations of matrix multiplication (e.g., a second iteration of matrix multiplication based on a first iteration of matrix multiplication, a final iteration of matrix multiplication based on a second-to-last iteration of matrix multiplication, or a culmination of previous iterations of matrix multiplication). In some cases, the query processing of QKV system 510 may include layer-by-layer normalization. In some cases, the query processing may include forward processing. The query processing may include a decoding function.As shown, query processing can return a response to the query. In some cases, one or more steps of query processing can be repeated (e.g., repeating matrix multiplications a certain number of iterations, e.g., 96 times, etc.).

[0118] Fig. 6 shows a flowchart illustrating an example method 600 in connection with the disclosed systems in accordance with the example implementations described herein. In some configurations, the method 600 may be performed by the query processing controller 140 of Fig. 1, the query processing controller 230 of Fig. 2, the PNM SSD SoC 305 from Fig. 3, the QKV system 410 from Fig. 4 and / or the query processing controller 515 of Fig. 5. In some configurations, the method 600 may be implemented in connection with the machine 105, components of the machine 105, or any combination thereof. The method 600 is only one implementation, and one or more operations of the method 600 may be rearranged, reordered, omitted, and / or otherwise modified, so that other implementations are possible and contemplated.

[0119] At 605, method 600 may include receiving attention data from a GPU processing device. For example, query processing controller 140 may receive attention data from a GPU processing device. The attention data may include QKV data of a query (e.g., vector Q1[N], vector K1[N], vector V1[N]), which may be based on attention data from a previous iteration.

[0120] At 610, method 600 may include processing key values ​​from the attention data with transposed query values ​​from the attention data. For example, query processing controller 140 may process key values ​​from the attention data with transposed query values ​​from the attention data.

[0121] At 615, method 600 may include determining a probability distribution of a result of the processing. For example, query processing controller 140 may determine a probability distribution of a result based on the processing.

[0122] At 620, method 600 may include generating an activation value (e.g., a set of activation values, an activation vector) based on the probability distribution, wherein the activation value indicates relevance between text units in a query associated with the attention data. For example, query processing controller 140 may generate an activation value based on the probability distribution, wherein the activation value indicates relevance between text units in a query associated with the attention data.

[0123] Fig. 7 shows a flowchart illustrating an example method 700 in connection with the disclosed systems in accordance with the example implementations described herein. In some configurations, the method 700 may be performed by the query processing controller 140 of Fig. 1, the query processing controller 230 of Fig. 2, the PNM SSD SoC 305 from Fig. 3, the QKV system 410 from Fig. 4 and / or the query processing controller 515 of Fig. 5. In some configurations, method 700 may be implemented in connection with machine 105, components of machine 105, or any combination thereof. Method 700 is only one implementation, and one or more operations of method 700 may be rearranged, reordered, omitted, and / or otherwise modified, so that other implementations are possible and contemplated.

[0124] At 705, method 700 may include receiving attention data from a GPU processing device. For example, query processing controller 140 may receive attention data from a GPU processing device.

[0125] At 710, method 700 may include processing key values ​​from the attention data with transposed query values ​​from the attention data. For example, query processing controller 140 may process key values ​​from the attention data with transposed query values ​​from the attention data.

[0126] At 715, method 700 may include determining a probability distribution of a result of the processing. For example, query processing controller 140 may determine a probability distribution of a result based on the processing.

[0127] At 720, method 700 may include generating an activation value (e.g., a set of activation values, an activation vector) based on the probability distribution, wherein the activation value indicates relevance between text units in a query associated with the attention data. For example, query processing controller 140 may generate an activation value based on the probability distribution, wherein the activation value indicates relevance between text units in a query associated with the attention data.

[0128] At 725, method 700 may include receiving at least one second activation value (e.g., a second activation vector) via a die-to-die (D2D) communication interface. For example, query processing controller 140 of a first PNM memory device may receive at least one second activation value from a second PNM memory device via a D2D communication interface.

[0129] At 730, method 700 may include forming a unified activation value based at least in part on a combination of multiple activation values. For example, query processing controller 140 may form a unified activation value based at least in part on a combination of the first activation value of the first PNM memory device and the second activation value of the second PNM memory device, wherein the D2D communication interface enables the first PNM memory device and the second PNM memory device to communicate independently of the GPU processing device or a host.

[0130] In the examples described herein, the configurations and operations are example configurations and operations and may include various additional configurations and operations not expressly depicted. In some examples, one or more aspects of the depicted configurations and / or operations may be omitted. In some embodiments, one or more of the operations may be performed by components other than those depicted herein. Additionally or alternatively, the sequential and / or temporal order of the operations may be varied.

[0131] Certain embodiments may be implemented in one or a combination of hardware, firmware, and software. Other embodiments may be implemented as instructions stored on a computer-readable storage medium that can be read and executed by at least one processor to perform the operations described herein. A computer-readable storage device may include any non-transferable storage mechanism for storing information in a form readable by a machine (e.g., a computer). A computer-readable storage device may include, for example, read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, and other storage devices and media.

[0132] The word "exemplary" is used herein to mean "as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. As used herein, the terms "computing device," "user device," "communications station," "station," "hands-free device," "mobile device," "wireless device," and "user equipment" (UE) refer to a wireless communications device, such as a cellular phone, smartphone, tablet, netbook, wireless terminal, laptop, femtocell, high data rate (HDR) subscriber station, access point, printer, point of sale, access terminal, or other personal communication system (PCS) device. The device may be either mobile or stationary.

[0133] As used in this document, the term "communicating" is intended to include sending or receiving, or both sending and receiving. This can be particularly useful in claims when describing the organization of data transmitted by one device and received by another, but only the functionality of one of those devices is required to infringe the claim. Similarly, the bidirectional exchange of data between two devices (both devices sending and receiving during the exchange) can be described as "communicating" when only the functionality of one of those devices is claimed. The term "communicating" as used herein with reference to a wireless communication signal includes sending the wireless communication signal and / or receiving the wireless communication signal.For example, a wireless communication unit capable of transmitting a wireless communication signal may include a wireless transmitter for transmitting the wireless communication signal to at least one other wireless communication unit and / or a wireless communication receiver for receiving the wireless communication signal from at least one other wireless communication unit.

[0134] Some embodiments may be used in connection with various devices and systems, such as a personal computer (PC), a desktop computer, a mobile computer, a laptop computer, a notebook computer, a tablet computer, a server computer, a handheld computer, a handheld device, a personal digital assistant (PDA), a handheld PDA device, an on-board device, an off-board device, a hybrid device, a vehicle device, a non-vehicle device, a mobile or portable device, a consumer device, a non-mobile or non-portable device, a wireless communication station, a wireless communication device, a wireless access point (AP), a wired or wireless router, a wired or wireless modem, a video device, an audio device, an audio-video (A / V) device, a wired or wireless network, a wireless network, a wireless video area network (WVAN),a local area network (LAN), a wireless LAN (WLAN), a personal area network (PAN), a wireless PAN (WPAN), and the like.

[0135] Some embodiments may be used in connection with one-way and / or two-way radio communication systems, cellular radiotelephone communication systems, a mobile telephone, a cellular telephone, a cordless telephone, a Personal Communication Systems (PCS) device, a PDA device including a wireless communication device, a mobile or portable Global Positioning System (GPS) device, a device including a GPS receiver or transceiver or chip, a device including an RFID element or an RFID chip, a multiple-input-multiple-output (MIMO) transceiver or device, a single-input-multiple-output (SIMO) transceiver or device, a multiple-input-single-output (MISO) transceiver or device, a device having one or more internal antennas and / or external antennas, Digital Video Broadcast (DVB) devices or systems, multi-standard radio devices or systems, a wired or wireless handheld device, e.g., a smartphone, a WAP (Wireless Application Protocol) device or similar.

[0136] Some embodiments may be used in connection with one or more types of wireless communication signals and / or systems that follow one or more wireless communication protocols, e.g. B. Radio frequency (RF), infrared (IR), frequency division multiplexing (FDM), orthogonal FDM (OFDM), time division multiplexing (TDM), time division multiplexing (TDMA), extended TDMA (E-TDMA), general packet radio service (GPRS), extended GPRS, code division multiple access (CDMA), wideband CDMA (WCDMA), CDMA 2000, single-carrier CDMA, multi-carrier CDMA, multi-carrier modulation (MDM), Discrete Multi-Tone (DMT), Bluetooth™, Global Positioning System (GPS), Wi-Fi, Wi-Max, ZigBee™, Ultra-Wideband (UWB), Global System for Mobile communication (GSM), 2G, 2.5G, 3G, 3.5G, 4G, fifth generation mobile networks (5G), 3GPP, Long Term Evolution (LTE), LTE advanced, Enhanced Data rates for GSM Evolution (EDGE), or something similar.Other embodiments may be used in various other devices, systems, and / or networks.

[0137] Although an example processing system has been described above, embodiments of the subject matter and the functional operations described herein may be implemented in other types of digital electronic circuits or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0138] Embodiments of the subject matter and operations described herein may be implemented in digital electronic circuits or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more components of computer program instructions encoded on a computer storage medium to be executed by, or to control the operation of, an information / data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated, propagating signal, e.g.,a machine-generated electrical, optical, or electromagnetic signal generated to encode information / data for transmission to a suitable receiving device for execution by an information / data processing device. A computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these. Although a computer storage medium is not a transmitted signal, a computer storage medium may be a source or destination of computer program instructions encoded in a man-made transmitted signal. The computer storage medium may also consist of or be included in one or more separate physical components or media (e.g., multiple CDs, floppy disks, or other storage devices).

[0139] The operations described herein may be implemented as operations performed by an information / data processing device on information / data stored on one or more computer-readable storage media or received from other sources.

[0140] The term "data processing device" encompasses all types of devices, apparatus, and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or several or combinations of the aforementioned devices. The device may contain special logic circuits, e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). In addition to hardware, the device may also contain code that creates an execution environment for the computer program in question, e.g., code representing processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more thereof. The device and the execution environment may implement various computer model infrastructures, e.g.,Web services, distributed computing and grid computing infrastructures.

[0141] A computer program (also called a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but does not have to, correspond to a file in a file system. A program may be stored in part of a file that also contains other programs or information / data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more components, subroutines, or pieces of code).A computer program may be provided for execution on one computer or on multiple computers located at one location or distributed across multiple locations and interconnected by a communications network.

[0142] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by acting on input information / data and producing outputs. Processors suitable for executing a computer program include, for example, both general-purpose and special-purpose microprocessors, as well as one or more processors of any type of digital computer. Generally, a processor receives instructions and information / data from read-only memory or random-access memory, or both. The essential elements of a computer are a processor for performing actions in accordance with instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, e.g.magnetic, magneto-optical, or optical disks, or is operatively coupled to receive information / data from or transmit information / data to one or more mass storage devices, or both. However, a computer need not have such devices. Devices suitable for storing computer program instructions and information / data include all forms of non-volatile memory, media, and storage devices, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or integrated with, special-purpose logic circuitry.

[0143] To enable user interaction, the embodiments described herein may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information / data to the user, as well as a keyboard and pointing device, such as a mouse or trackball, for the user to provide input to the computer. Other types of devices may also be used to enable user interaction; for example, feedback to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and user input may be received in any form, including auditory, voice, or tactile input.In addition, a computer can interact with a user by sending and receiving documents to and from a device used by the user, for example, by sending web pages to a web browser on a client device of the user in response to requests received from the web browser.

[0144] Embodiments of the subject matter described herein may be implemented in a computer system that includes a back-end component, e.g., an information / data server, or a middleware component, e.g., an application server, or a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital information / data communication, for example, by a communications network. Examples of communications networks include a local area network ("LAN") and a wide area network ("WAN"), an internetwork (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0145] The computer system may include clients and servers. A client and a server are typically remote from each other and typically interact over a communications network. The relationship between client and server arises because computer programs run on the respective computers and have a client-server relationship with each other. In some embodiments, a server transmits information / data (e.g., an HTML page) to a client device (e.g., for displaying information / data and receiving user input from a user interacting with the client device). Information / data generated on the client device (e.g., a result of user interaction) may be received by the client device on the server.

[0146] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any embodiment or what may be claimed, but rather as descriptions of features specific to particular embodiments. Certain features described herein in connection with individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in connection with a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination.Furthermore, although features are described above as operating in certain combinations and even originally claimed as such, in some cases one or more features from a claimed combination may be taken out of the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0147] Although the operations are illustrated in the drawings in a particular order, this should not be understood to mean that these operations must be performed in the order shown or in sequential order, or that all of the illustrated operations must be performed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be understood to mean that such separation is required in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0148] Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the acts recited in the claims may be performed in a different order and still achieve the desired results. Moreover, the processes depicted in the accompanying figures do not necessarily require the depicted order or the sequential order to achieve desirable results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0149] Many modifications and other examples described herein will occur to those skilled in the art to which these embodiments belong when considering the teachings presented in the foregoing descriptions and the accompanying drawings. Therefore, it is to be understood that the embodiments are not limited to the specific embodiments disclosed herein, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used only in a generic and descriptive sense and not for purposes of limitation. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 608,821

[0001]

Claims

[1] A method for query processing, the method comprising: Receiving data in a first near-memory processing (PNM) storage device; processing, at the first PNM storage device, first values ​​from the data with transposed second values ​​from the data; Determining, at the first PNM storage device, a probability distribution of a result of processing the first values ​​from the data with the transposed second values ​​from the data; and Generating, at the first PNM storage device, an activation value based on the probability distribution, the activation value indicating a correlation between text units in a query associated with the data. [2] The method of claim 1, wherein the first PNM storage device includes a memory that stores key-value data from the data. [3] The method of claim 2, wherein the memory and a processor of the first PNM memory device comprise an integrated circuit based on the memory being stacked above the processor and the memory being communicatively coupled to the processor. [4] The method of claim 1, further comprising: Receiving at least a second activation value from a second PNM memory device via a die-to-die (D2D) communication interface; and Forming a third activation value based at least in part on a combination of the activation value of the first PNM memory device and the second activation value of the second PNM memory device, wherein the D2D communication interface enables the first PNM memory device and the second PNM memory device to communicate. [5] The method of claim 1, further comprising receiving a trigger from the processing device based on an indication of a previous operation, the trigger including at least one of the following information: a user identification or a layer number information associated with the data. [6] The method of claim 1, wherein the data is a part of the multi-head attention data distributed across multiple PNM storage devices. [7] The method of claim 1, wherein the data includes a portion of query attention data from a first attention layer, a portion of key attention data from a second attention layer, and a portion of value attention data from a third attention layer. [8] The method of claim 1, wherein the data is based on an iteration of activation values ​​generated based on partial outputs generated by multiple PNM memory devices combined into a unified output. [9] The method of claim 8, wherein the plurality of PNM storage devices comprise an array of solid-state drives. [10] The method of claim 1, wherein the first PNM storage device is a system-on-chip die that includes a solid-state drive and at least one processor that performs query processing. [11] The method of claim 1, wherein the first PNM memory device is communicatively coupled to the processing device via a high bandwidth expansion bus. [12] The method of claim 1, wherein the processing device comprises at least one graphics processing unit (GPU) communicatively coupled to a high-bandwidth memory. [13] The method of claim 2, wherein the data contains attention data, the first values ​​contain key values, the second values ​​contain query values, and the key-value data has a key-value matrix. [14] Query processing system, the query processing system comprising: a processing device communicatively coupled to a first near-memory (PNM) storage device, the processing device transferring data to the first PNM storage device; and the plurality of near-memory processing (PNM) memory devices, wherein the first PNM memory device is configured to: to process first values ​​from the data with transposed second values ​​from the data; to determine a probability distribution of a processing result; and generate an activation value based on the probability distribution, where the activation value indicates a correlation between text units in a query associated with the data. [15] The query processing system of claim 14, wherein the first PNM storage means includes a memory that stores key-value data from the data. [16] The query processing system of claim 15, wherein the memory and a processor of the first PNM memory device comprise an integrated circuit based on the memory being stacked on the processor and the memory being communicatively coupled to the processor. [17] The query processing system of claim 14, wherein the first PNM storage device is configured to: Receiving at least a second activation value from a second PNM memory device via a die-to-die communication interface (D2D); and Forming a unified activation value based at least in part on a combination of the activation value of the first PNM memory device and the second activation value of the second PNM memory device, wherein the D2D communication interface enables the first PNM memory device and the second PNM memory device to communicate. [18] A non-transitory computer-readable medium storing code, the code comprising instructions executable by a processor of a first near-memory (PNM) storage device to: to process first values ​​from data with transposed query values ​​from the data at the first PNM storage device; to determine a probability distribution of a result of the processing at the first PNM storage device; and generating at the first PNM storage device an activation value based on the probability distribution, the activation value indicating a correlation between text units in a query associated with the data. [19] The non-transitory computer-readable medium of claim 18, wherein: the first PNM storage device includes a memory that stores key-value data from the data, and the memory and a processor of the first PNM memory device comprise an integrated circuit based on the memory being stacked on the processor and the memory being communicatively connected to the processor.

Images