In-store attention engine

Abstract

The invention relates to an in-memory attention engine. A computing system including an attention engine is disclosed. The attention engine is an in-memory computing module that can be used to accelerate attention operations. The attention engine includes dot product circuitry and multiplier circuitry for together performing matrix generation and matrix multiplication in an analog domain. Performing the matrix multiplication in the analog domain may be faster and / or consume less power than performing the matrix multiplication in the digital domain.

Description

Background Art

[0001] Neural networks are a class of machine learning algorithms inspired by the structure and function of the human brain. They consist of interconnected nodes organized into layers, where each node takes input data, processes it, and passes output to the next layer. These networks are trained on large amounts of data by adjusting the strength of connections between nodes (e.g., weights). As a result, neural networks can learn complex patterns and representations from data, enabling them to excel in tasks such as natural language processing, image recognition, and decision making. A notable advancement in neural networks is the transformer architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Aspects of the present disclosure are best understood from the following detailed description when read with the accompanying figures.

[0003] Figure 1 is a block diagram of a computing system according to some implementations.

[0004] Figure 2 is a block diagram of some components of an attention engine according to some embodiments.

[0005] Figure 3 is a schematic diagram of an attention engine according to some embodiments.

[0006] Figure 4 is a simplified diagram of a programmable crossbar array for an attention engine, according to some embodiments.

[0007] Figure 5 is a block diagram of some components of an attention engine according to some embodiments.

[0008] Figure 6 is a simplified diagram of an attention engine programming method according to some embodiments.

[0009] Figure 7 is a simplified diagram of a transformation method according to some embodiments.

[0010] Corresponding numerals and symbols in the different figures generally refer to corresponding parts unless otherwise indicated.The figures are drawn to clearly illustrate the relevant aspects of the disclosure and are not necessarily drawn to scale. DETAILED DESCRIPTION

[0011] The following disclosure provides many different examples for implementing different features. Specific examples of components and arrangements are described below to simplify the disclosure. Of course, these are merely examples and are not intended to be limiting.

[0012] One type of neural network is a large language model (LLM). An LLM can be trained to predict the token of a segment. The predicted token may be the next token of the segment or may be the missing token of the segment. Some types of LLMs, such as generative pre-trained transformers (GPTs), use transformer architectures. In a transformer architecture, an LLM is organized into multiple layers, each of which includes a transformer. The transformer obtains an input sequence, adds information to help clarify the meaning of the elements in the input sequence, and generates an output sequence (also called a hidden state sequence). The hidden state sequence generated by the transformer is fed into a corresponding subsequent transformer. The hidden state sequence generated by the final layer is used to predict the token of the segment.

[0013] The Transformer architecture uses an attention mechanism. Each Transformer performs an attention operation. The attention operation computes the relationship between the elements of a sequence. A Transformer uses an attention operation to add information to its input sequence when generating its sequence of hidden states. One attention mechanism that can be used in the Transformer architecture is Scaled Dot Product Attention (SDPA). The performance of a neural network using SDPA depends on the length of the input sequence. For longer input sequence lengths, the SDPA operation can take up most of the processing time of the neural network. Therefore, increasing the speed of the SDPA operation can reduce the processing time of the neural network.

[0014] In an example implementation, the present disclosure discloses a computing system including an attention engine. The attention engine is an in-memory computing module that can be used to accelerate SDPA operations of the computing system. The attention engine includes a dot product circuit and a multiplier circuit that are used together to perform matrix multiplication for SDPA operations.

[0015] In an example embodiment, the dot product circuit includes a dot product engine (DPE) that is part of a programmable crossbar array and is used to generate matrices for attention operations. The multiplier circuit is coupled to the dot product engine and can be used to multiply the matrices generated by the dot product engine. Both the matrix generation and the matrix multiplication can be performed in the analog domain.

[0016] In an example embodiment, the digital input of the attention engine can be converted to an analog input signal. The analog input signal is input to a dot product circuit, which, together with a multiplier circuit, performs matrix generation and multiplication in the analog domain and produces an analog output signal. The analog output signal is then converted back to a digital output of the attention engine. Performing matrix multiplication in the analog domain can be faster and / or consume less power than performing matrix multiplication in the digital domain. Accordingly, the performance of the SDPA operation can be improved.

[0017] In an example implementation, the inputs for SDPA operations include queries, keys, and values. The queries, keys, and values ​​are generated from the input sequence of the transformer. The outputs of SDPA can be calculated according to equations (1) to (4).

[0018]

[0019] Q=XW Q (2)

[0020] K=XW K (3)

[0021] V=XW V (4)

[0022] In equations (1) to (4), X is the input matrix. Each element of the input sequence is a vector, and each row of the input matrix X is an element of the input sequence. Additionally, d k is the length of each element of the input sequence, W Q is the query weight matrix, W K is the key weight matrix, and W V is the value weight matrix. By making X and W Q Multiply to calculate the query matrix Q. By making X and W K Multiply to calculate the key matrix K. By making X and W V Multiply to calculate the value matrix V. Finally, A is the resulting attention matrix. The attention engine of the present disclosure can be used to perform at least some of the matrix multiplication operations in equations (1) to (4) and perform them in the analog domain rather than the digital domain. Therefore, the speed of SDPA operation can be improved.

[0023] Figure 1 1 is a block diagram of an example computing system 100 that can be used to operate an LLM based on the converter architecture as previously described. The computing system 100 can be implemented in an electronic device. Examples of electronic devices include servers, desktop computers, portable computers, mobile devices, gaming systems, and the like.

[0024] The computing system 100 can be used in any data processing scenario, including standalone hardware, mobile applications, or a combination thereof. Further, the computing system 100 can be used in a computing network, such as a public cloud network, a private cloud network, a hybrid cloud network, other forms of networks, or a combination thereof. In one example, the method provided by the computing system 100 is provided as a service by, for example, a third party over a network. The computing system 100 can be implemented on one or more hardware platforms, wherein the modules in the system can be executed on one or more platforms. Such modules can run on various forms of cloud technologies and hybrid cloud technologies, or be provided as software as a service that can be implemented in and outside the cloud.

[0025] To achieve its desired functionality, computing system 100 includes various hardware components. These hardware components may include processor 102, one or more interfaces 104, memory 106, and attention engine 200. The hardware components may be interconnected via multiple buses and / or network connections. In one example, processor 102, (multiple) interfaces 104, memory 106, and attention engine 200 may be communicatively coupled via bus 108.

[0026] The processor 102 retrieves the executable code from the memory 106 and executes the executable code. The executable code, when executed by the processor 102, may cause the processor 102 to implement any of the functions described herein. The processor 102 may be a microprocessor, an application specific integrated circuit, a microcontroller, or the like.

[0027] The interface(s) 104 enable the processor 102 to interface with various other hardware elements external and internal to the computing system 100. For example, the interface(s) 104 may include interface(s) with input / output devices such as, for example, a display device, a mouse, a keyboard, etc. Additionally or alternatively, the interface(s) 104 may include interface(s) with external storage devices or with a plurality of network devices such as servers, switches and routers, client devices, other types of computing devices, and combinations thereof.

[0028] The memory 106 may include various types of memory modules, including volatile and non-volatile memory. For example, the memory 106 may include random access memory (RAM), read-only memory (ROM), hard disk drive (HDD), etc. The memory 106 may include a non-transitory computer-readable medium storing instructions for execution by the processor 102. One or more modules within the computing system 100 may be partially or completely embodied as software and / or hardware for performing any of the functions described herein. Different types of memory may be used for different data storage needs. For example, in some examples, the processor 102 may boot from ROM, maintain non-volatile storage in the HDD, and execute program code stored in RAM.

[0029] The attention engine 200 is an accelerator for performing attention operations, which is separate from the processor 102 and from the memory 106. The attention engine 200 can be an in-memory computing module that can be used to accelerate SDPA operations. The attention engine 200 includes one or more dot product circuits and one or more multiplier circuits that are collectively configured to perform SDPA operations in the analog domain. The (multiple) dot product circuits each include multiple dot product engines. The dot product engines are implemented using a programmable crossbar array for performing matrix multiplication. The programmable crossbar array includes multiple programmable non-volatile analog elements that work together within the array to perform weighted sums on multiple inputs. The programmable crossbar array can be used as an accelerator, where the array performs multiple functions faster than software running on a more general processing device.

[0030] Figure 2 is a block diagram of some components of the attention engine 200 according to some embodiments. The attention engine 200 is used to perform matrix generation and multiplication operations. Specifically, the attention engine 200 can be used to generate a query matrix Q and a key matrix K using equations (2) and (3), respectively, and then calculate QK in equation (1) by multiplying the query matrix Q with the transpose of the key matrix K. T Attention engine 200 may include digital-to-analog converter 202 , dot product circuit 204 , multiplier circuit 206 , and analog-to-digital converter 208 . Figure 2 The signal flow in the attention engine 200 is illustrated, where signals in the analog domain are shown with solid lines.

[0031] The DAC 202 receives a digital input matrix X from the processor 102 (see Figure 1 ) and convert the digital input matrix X into an analog input matrix X. Each row of the digital input matrix X is an element of the input sequence of the converter. Each element of the analog input matrix X is an analog signal corresponding to an element of the digital input matrix X of the converter. Specifically, the voltage of the element of the analog input matrix X can be proportional to the digital value of the corresponding element of the digital input matrix X. Example digital-to-analog converters include resistor strings, delta-sigma modulators, etc. The digital-to-analog converter 202 can include multiple converter modules (e.g., one converter module for each row of the digital input matrix X), or one converter module with multiple channels.

[0032] The dot product circuit 204 is coupled to the digital-to-analog converter 202. The dot product circuit 204 receives the analog input matrix X from the digital-to-analog converter 202 and generates an analog query matrix Q and an analog key matrix K. Each element of the analog query matrix Q is an analog signal corresponding to an element of the digital query matrix Q (see equation (2)). Each element of the analog key matrix K is an analog signal corresponding to an element of the digital key matrix K (see equation (3)). The dot product circuit 204 may include a query dot product engine 210 and a key dot product engine 212. Subsequently, the query matrix Q is used to generate the analog query matrix Q and the key key matrix K. Figure 3 An example implementation of the dot product circuit 204 is described.

[0033] Each of the query dot product engines 210 stores and queries the weight matrix W Q (See equation (2)). Specifically, each of the query dot product engines 210 stores the query weight matrix W Q The query dot product engine 210 is configured to perform the simulation by comparing the simulated input matrix X with the query weight matrix W stored in the query dot product engine 210. Q Multiply to generate the simulated query matrix Q.

[0034] Each of the key dot product engines 212 stores the key weight matrix W K (See equation (3)). Specifically, each of the key dot product engines 212 stores the key weight matrix W K The key dot product engine 212 is configured to generate a plurality of analog input matrices X and a plurality of key weight matrices W stored in the key dot product engine 212. K Multiply to generate the simulated key matrix K.

[0035] The multiplier circuit 206 is coupled to the dot product circuit 204. The multiplier circuit 206 is configured to calculate the analog output matrix QK by multiplying the analog query matrix Q with the transpose of the analog key matrix K. T . Analog output matrix QK T Each element of is an analog signal corresponding to an element of the matrix that would be produced if the digital query matrix Q was multiplied by the transpose of the digital key matrix K in the digital domain. Thus, the multiplier circuit 206 performs the equivalent of matrix multiplication in the digital domain in the analog domain. Figure 3 An example implementation of multiplier circuit 206 is described.

[0036] The analog-to-digital converter 208 is coupled to the multiplier circuit 206. The analog-to-digital converter 208 receives the analog output matrix QK T And the simulation output matrix QK T Convert to digital output matrix QK T. Digital output matrix QK T Each element of is the result of taking the dot product of a row of the digital query matrix Q and the corresponding row of the digital key matrix K. The digital output matrix QK T can be output to the processor 102 (see Figure 1 ) or can be output to a digital circuit (which is then Figure 5 The digital output matrix QK can then be used T Other appropriate operations from equation (1) are performed to calculate the output sequence of the converter.Example analog-to-digital converters include successive approximation converters, sigma-delta modulators, and the like.

[0037] Figure 3 2 is a schematic diagram of an attention engine 200 according to some embodiments. Example implementations of the dot product circuit 204 and the multiplier circuit 206 are shown, but they can be implemented in other ways. The attention engine 200 accepts a digital input matrix X. Each row of the digital input matrix X is an element of the input sequence of the transformer. The attention engine 200 includes a query dot product engine 210 and a key dot product engine 212 for each element of the input sequence (e.g., each row of the input matrix X). The maximum supported length of the input sequence (e.g., the maximum number of rows of the digital input matrix X) is determined by the number of components of the attention engine 200. For simplicity of illustration, Figure 3 The attention engine 200 supports an input sequence having two elements. It should be understood that the attention engine can support longer input sequences by having more components described subsequently (connected together as appropriate).

[0038] The digital-to-analog converter 202 receives the elements of the input sequence and converts them into analog signals. As previously described, each element of the input sequence is a vector. Therefore, the digital-to-analog converter 202 generates a vector x1 of analog signals corresponding to the first element of the input sequence, and generates a vector x2 of analog signals corresponding to the second element of the input sequence.

[0039] The dot product circuit 204 includes a plurality of crossbar arrays 400 coupled to the digital-to-analog converter 202. The crossbar arrays 400 include a first programmable crossbar array 400A (for the first element of the input sequence) and a second programmable crossbar array 400B (for the second element of the input sequence). The first programmable crossbar array 400A includes a first query dot product engine 210A and a first key dot product engine 212A. The second programmable crossbar array 400B includes a second query dot product engine 210B and a second key dot product engine 212B. The programmable crossbar array 400 is used to perform vector-matrix multiplication in the analog domain. Specifically, the programmable crossbar array 400 includes a plurality of programmable elements within the array that work together to perform a weighted sum on the output of the digital-to-analog converter 202. Turning briefly to Figure 4 , an example implementation of a programmable crossbar switch array 400 will be described.

[0040] Figure 4 4 is a simplified diagram of a programmable crossbar array 400 for an attention engine according to some embodiments. The programmable crossbar array 400 includes a plurality of input electrodes 402, a plurality of output electrodes 404 (including output electrodes 404A and output electrodes 404B), and a plurality of programmable elements 406 (including programmable elements 406A and programmable elements 406B). The input electrodes 402 are arranged in rows and the output electrodes 404 are arranged in columns. Each programmable element 406 is positioned at the intersection or junction of an input electrode 402 and an output electrode 404. As input, the programmable crossbar array 400 acquires a vector of analog signals (on the input electrodes 402).

[0041] Programmable element 406 is a circuit element whose conductivity is programmable. Programmable element 406 is a non-volatile analog device that can be suitable for storing multiple data bits. An example of a programmable element is a memristor including a dielectric layer (e.g., an oxide layer) between two metal layers. When programmable element 406 is a memristor, programmable crossbar switch array 400 is a memristor array. Other examples of programmable elements include multi-bit flash memory cells, resistive random access memory (ReRAM) cells, phase change random access memory (PCRAM) cells, magnetoresistive random access memory (MRAM) cells, electrochemical random access memory (ECRAM) cells, and the like.

[0042] The programmable crossbar switch array 400 may also include other peripheral circuits (not shown separately) associated with the programmable crossbar switch array 400 when used as a storage device. For example, the programmable crossbar switch array 400 may include a driver connected to the input electrode 402. The address decoder may be used to select the input electrode 402 and activate the driver corresponding to the selected input electrode 402. The driver for the selected input electrode 402 may drive the corresponding input electrode 402 with different voltages, which correspond to the process of vector matrix multiplication or setting the resistance value in the programmable element 406 of the programmable crossbar switch array 400. Similar driver and decoder circuits may be included for the output electrode 404. The control circuit may also be used to control the voltage application at the input portion of the programmable crossbar switch array 400. The input signals to the input electrode 402 and the output electrode 404 are analog signals. The peripheral circuits described above may be made using semiconductor processing technology in the same integrated structure or semiconductor die as the programmable crossbar switch array 400.

[0043] The programmable crossbar array 400 includes a query dot product engine 210 and a key dot product engine 212. A first subset of output electrodes 404A and a first subset of programmable elements 406A are used for the query dot product engine 210. A second subset of output electrodes 404B and a second subset of programmable elements 406B are used for the key dot product engine 212. The programmable crossbar array 400 includes N input electrodes 402, M output electrodes 404A, and M output electrodes 404B. As described in more detail below, there are two main operations that occur during the operation of the programmable crossbar array 400. The first operation is to program the programmable elements 406 in the programmable crossbar array 400 so as to map mathematical values ​​in an N×M matrix to programmable elements 406A for the query dot product engine 210 and map mathematical values ​​in another N×M matrix to programmable elements 406B for the key dot product engine 212. The second operation is a dot product or vector-matrix multiplication operation. In this operation, an input voltage is applied to input electrode 402 and an output current is obtained from output electrode 404, which corresponds to the result of multiplying an N×1 vector by an N×M matrix. The input voltage is below the threshold of the programming voltage of programmable element 406, so the resistance value of the programmable element in programmable crossbar switch array 400 is not changed during the vector-matrix multiplication operation.

[0044] Vector-matrix multiplication may be performed by the programmable crossbar array 400 by simultaneously applying a set of voltages along the input electrodes 402 of the programmable crossbar array 400 and collecting the current through the output electrodes 404. The signals generated on the output electrodes 404 are weighted by the corresponding conductivity of the programmable elements 406 at the intersections of the output electrodes 404 with the input electrodes 402, and the weighted sum is reflected in the current at the output electrodes 404. Thus, the relationship between the voltage at the input electrode 402 and the current at the output electrode 404A is represented by the vector-matrix multiplication of the input vector and the N×M matrix determined by the conductivity of the programmable element 406A as the query dot product engine 210. Similarly, the relationship between the voltage at the input electrode 402 and the current at the output electrode 404B is represented by the vector-matrix multiplication of the input vector and the N×M matrix determined by the conductivity of the programmable element 406B as the key dot product engine 212.

[0045] The programmable crossbar array 400 can be programmed to store an N×M matrix for the query dot product engine 210 and the key dot product engine 212 by modifying the conductivity of the programmable element 406. The conductivity of the programmable element 406 is a value corresponding to the N×M matrix. The conductivity of the programmable element 406 can be modified by applying a voltage across the programmable element 406 using the input electrode 402, the output electrode 404, and the corresponding voltage driver. The voltage difference applied across the programmable element 406 generally determines the resulting conductivity of the programmable element 406. The programming process can be performed row by row.

[0046] Return to Figure 3 , the operation of the programmable crossbar switch array 400 will be described. Query weight matrix W Q The key weight matrix W is stored in the programmable elements of the query dot product engine 210, which are used to calculate the corresponding rows of the query matrix Q (see equation (2)). K Stored in the programmable elements of the key dot product engine 212, these programmable elements are used to calculate the corresponding rows of the key matrix K (see equation (3)).

[0047] The input of the first programmable crossbar switch array 400A (eg, input electrode 402) is connected to a subset of the outputs of the digital-to-analog converter 202. The first query dot product engine 210A and the first key dot product engine 212A respectively connect the query weight matrix W Q and key weight matrix W K The corresponding values ​​are stored in their programmable elements 406. The vector x1 output from the DAC 202 is combined with the query weight matrix W in the first query dot product engine 210A. Q and the key weight matrix W in the first key dot product engine 212AK The multiplication is performed in memory by the first programmable crossbar array 400A. The first programmable crossbar array 400A outputs two vectors of analog signals: by combining the vector x1 with the query weight matrix W stored in the first query dot product engine 210A. Q The vector q1 of the analog signal generated by multiplication (on the output electrode 404A), and the key weight matrix W stored in the first key dot product engine 212A by comparing the vector x1 with the key weight matrix W K The vector k1 of the analog signal generated by multiplication (on output electrode 404B). Vector q1 corresponds to the first row of the query matrix Q (see equation (2)), while vector k1 corresponds to the first row of the key matrix K (see equation (3)). Vector q1 and vector k1 have the same length. The length depends on the query weight matrix W Q and key weight matrix W K The number of columns.

[0048] The input of the second programmable crossbar switch array 400B (eg, input electrode 402) is connected to a subset of the outputs of the digital-to-analog converter 202. The second query dot product engine 210B and the second key dot product engine 212B respectively convert the query weight matrix W Q and key weight matrix W K The corresponding values ​​are stored in their programmable elements 406. The vector x2 output from the DAC 202 is combined with the query weight matrix W in the second query dot product engine 210B. Q and the key weight matrix W in the second key dot product engine 212B K The second programmable crossbar array 400B performs the multiplication in memory. The second programmable crossbar array 400B outputs two vectors of analog signals: by combining the vector x2 with the query weight matrix W stored in the second query dot product engine 210B. Q The vector q2 of analog signals generated by multiplication (on output electrode 404A) and the key weight matrix W stored in the second key dot product engine 212B are obtained by combining the vector x2 with the key weight matrix W stored in the second key dot product engine 212B. K The vector k2 of the analog signal generated by multiplication (on output electrode 404B). Vector q2 corresponds to the second row of the query matrix Q (see equation (2)), while vector k2 corresponds to the second row of the key matrix K (see equation (3)). Vector q2 and vector k2 have the same length. The length depends on the query weight matrix W Q and key weight matrix W K The number of columns.

[0049] Same query weight matrix W Q stored in both the first query dot product engine 210A and the second query dot product engine 210B, while the same key weight matrix WK =W is stored in both the first key dot product engine 212A and the second key dot product engine 212B. Therefore, the first input sequence element and the query weight matrix W Q and key weight matrix W K The second input sequence element can be multiplied with the query weight matrix W Q and key weight matrix W K Each matrix multiplication occurs simultaneously. Each matrix multiplication is performed in the analog domain by a corresponding programmable crossbar switch array 400.

[0050] The multiplier circuit 206 includes a plurality of current multipliers 302 and a plurality of current summers 304. In an embodiment, the multiplier circuit 206 is a CMOS circuit. Each current multiplier 302 is connected to an output of the query dot product engine 210 and to an output of the key dot product engine 212 (e.g., two output electrodes 404). Specifically, different subsets of the current multipliers 302 are coupled to different subsets of the output electrodes 404 of the query dot product engine 210 and to different subsets of the output electrodes 404 of the key dot product engine 212. The current multipliers 302 can be based on bipolar junction transistors, MOSFET transistors, etc. Examples of current multipliers include Gilbert cells, etc. The outputs of multiple groups of current multipliers 302 are connected to corresponding current summers 304, which sum the currents from the current multipliers 302. Each current summer 304 can be a common node to which multiple current multipliers 302 are connected.

[0051] The first subset of current multipliers 302A receives a vector q1 of analog signals and a vector k1 of analog signals. The corresponding elements of vector q1 and vector k1 are fed into the corresponding current multiplier 302A. Therefore, the output of each current multiplier 302A is generated by multiplying the corresponding element of vector q1 with the corresponding element of vector k1. For example, the first element of vector q1 is multiplied with the first element of vector k1 using the first current multiplier 302A. The output of current multiplier 302A is fed into current summer 304A, which sums the currents from current multipliers 302A. Therefore, the output of current summer 304A is a value corresponding to the dot product of acquisition vector q1 (e.g., the first row of query matrix Q) and vector k1 (e.g., the first row of key matrix K).

[0052] The second subset of current multipliers 302B receives a vector q2 of analog signals and a vector k1 of analog signals. The corresponding elements of vector q2 and vector k1 are fed into the corresponding current multiplier 302B. Therefore, the output of each current multiplier 302B is generated by multiplying the corresponding element of vector q2 with the corresponding element of vector k1. For example, the first current multiplier 302B is used to multiply the first element of vector q2 with the first element of vector k1. The output of current multiplier 302B is fed into current summer 304B, which sums the currents from current multipliers 302B. Therefore, the output of current summer 304B is a value corresponding to the dot product of acquisition vector q2 (e.g., the second row of query matrix Q) and vector k1 (e.g., the first row of key matrix K).

[0053] The third subset of current multipliers 302C receives a vector q2 of analog signals and a vector k2 of analog signals. The corresponding elements of vector q2 and vector k2 are fed into the corresponding current multipliers 302C. Therefore, the output of each current multiplier 302C is generated by multiplying the corresponding element of vector q2 with the corresponding element of vector k2. For example, the first current multiplier 302C is used to multiply the first element of vector q2 with the first element of vector k2. The output of current multipliers 302C is fed into a current summer 304C, which sums the currents from current multipliers 302C. Therefore, the output of current summer 304C is a value corresponding to the dot product of acquisition vector q2 (e.g., the second row of query matrix Q) and vector k2 (e.g., the second row of key matrix K).

[0054] The fourth subset of current multipliers 302D receives a vector q1 of analog signals and a vector k2 of analog signals. The corresponding elements of vector q1 and vector k2 are fed into the corresponding current multiplier 302D. Therefore, the output of each current multiplier 302D is generated by multiplying the corresponding element of vector q1 with the corresponding element of vector k2. For example, the first current multiplier 302D is used to multiply the first element of vector q1 with the first element of vector k2. The output of current multiplier 302D is fed into current summer 304D, which sums the currents from current multipliers 302D. Therefore, the output of current summer 304D is a value corresponding to the dot product of acquisition vector q1 (e.g., the first row of query matrix Q) and vector k2 (e.g., the second row of key matrix K).

[0055] The attention engine 200 further includes a current-to-voltage converter 306. The current-to-voltage converter 306 is connected to the output of the multiplier circuit 206, for example, to the current summer 304. Each current-to-voltage converter 306 is connected to a corresponding current summer 304 and converts the output of the current summer 304 into a corresponding voltage. The current-to-voltage converter 306 can be based on an operational amplifier. Examples of current-to-voltage converters include transimpedance amplifiers, etc. The output of the current-to-voltage converter 306 is coupled to the input of the analog-to-digital converter 208, which converts the analog signal from the current-to-voltage converter 306 into a digital output matrix QK T The outputs of the current summers 304A and 304B are converted by the current-to-voltage converters 306A and 306B into a digital output matrix QK T The outputs of the current summators 304C and 304D are converted into a digital output matrix QK by the current-to-voltage converters 306C and 306D. T The second line of .

[0056] The components of attention engine 200 (as described above) may be fabricated in the same semiconductor die using semiconductor processing techniques. For example, dot product circuit 204 and multiplier circuit 206 may be part of the same semiconductor die.

[0057] The attention engine 200 can be divided into Figure 3 For example, a single query dot product engine can be used to store the query weight matrix W Q A copy of , and each output of the query dot product engine can be connected to more than one current multiplier 302. Similarly, a single key dot product engine can be used to store the key weight matrix W K and each output of the key dot product engine can be connected to more than one current multiplier 302 .

[0058] The digital output matrix QK can then be used T Other appropriate operations from equation (1) are performed to compute the output sequence of the transformer. The attention engine 200 may include additional features. In some embodiments, the attention engine 200 includes features for performing additional matrix generation and multiplication operations.

[0059] Figure 5 is a block diagram of some components of the attention engine 200 according to some embodiments. The attention engine 200 includes Figure 2, and further comprising components for performing the additional operations in equations (1) to (4). The attention engine 200 may be configured to generate a value matrix V using equation (4), and then calculate an attention matrix A using equation (1). The attention engine 200 may further include a digital circuit 502, a digital-to-analog converter 504, a digital-to-analog converter 506, a dot product circuit 508, a multiplier circuit 510, and an analog-to-digital converter 512. Figure 5 The signal flow in the attention engine 200 is illustrated, where signals in the analog domain are shown with solid lines and signals in the digital domain are shown with dashed lines.

[0060] The digital circuit 502 is connected to the analog-to-digital converter 208. The digital circuit 502 receives the digital output matrix QK from the analog-to-digital converter 208. T Next, digital circuit 502 calculates To obtain the SoftMax matrix. This calculation is performed in the digital domain. The digital circuit 502 may include an integrated circuit or the like.

[0061] The digital-to-analog converter 504 receives the data from the digital circuit 502 as a calculation The result of the SoftMax operation can then be converted back to the analog domain using a digital-to-analog converter 504. The output of the digital-to-analog converter 504 can be referred to as an analog SoftMax matrix.

[0062] In line with previous Figure 2 In a similar manner to the described DAC 202, DAC 506 is coupled from processor 102 (see Figure 1 ) receives a digital input matrix X and converts the digital input matrix X into an analog input matrix X. The digital-to-analog converter 506 may be similar to the digital-to-analog converter 202 .

[0063] The dot product circuit 508 is coupled to the digital-to-analog converter 506. The dot product circuit 508 receives the analog input matrix X from the digital-to-analog converter 506 and generates an analog value matrix V. Each element of the analog value matrix V is an analog signal corresponding to an element of the digital value matrix V (see equation (4)). The dot product circuit 508 may include a value dot product engine 214. Figure 3 In a manner similar to that described above, the value dot product engine 214 may be implemented using a programmable crossbar array 400 .

[0064] Each of the value dot product engines 214 stores a value weight matrix W V (See equation (4)). Specifically, each of the value dot product engines 214 stores the value weight matrix W VThe value dot product engine 214 is configured to generate a plurality of analog input matrix X and a plurality of weight matrix W stored in the value dot product engine 214. V Multiply to generate the simulation value matrix V.

[0065] The multiplier circuit 510 is coupled to the dot product circuit 508 and the digital-to-analog converter 504. The multiplier circuit 510 is configured to compute the analog attention matrix A by multiplying the analog value matrix V (from the dot product circuit 508) with the analog SoftMax matrix (from the digital-to-analog converter 504). Thus, the multiplier circuit 510 performs in the analog domain the equivalent of matrix multiplication in the digital domain. The multiplier circuit 510 may be implemented in the same manner as previously described for Figure 3 The previous implementation described. In an implementation, the multiplier circuit 510 is a CMOS circuit.

[0066] The analog-to-digital converter 512 is coupled to the multiplier circuit 510. The analog-to-digital converter 512 receives the analog attention matrix A and converts the analog attention matrix A into a digital attention matrix A. The digital attention matrix A can be output to the processor 102 (see Figure 1 ). The digital attention matrix A represents the relationship between the elements of the input sequence. The processor 102 can then use this relationship to generate a hidden state sequence for the transformer.

[0067] Figure 6 600 is a simplified diagram of an attention engine programming method 600 according to some embodiments. The attention engine programming method 600 is executed by the inference engine during the training of the LLM. Any processor can be used to execute the inference engine. For example, the attention engine programming method 600 can be executed by the processor 102 (see Figure 1 ) is executed. In some embodiments, the LLM is trained offline by another processor. The weights generated by the training are then programmed into the attention engine 200 for use in the computing system 100 (see Figure 1 ). The attention engine can be Figure 2 Attention Engine 200 or Figure 5 Attention Engine 200.

[0068] The processor performs step 602 of analyzing the learning data. The learning data may be previously known data. The processor performs step 604 of programming the dot product engine of the attention engine 200 based on the analysis of the learning data. Specifically, the weight matrix W is queried. Q , key weight matrix W K Sum value weight matrix W Vmay be calculated by the processor and then respectively stored in the query dot product engine 210, the key dot product engine 212, and the value dot product engine 214. The dot product engines may be programmed in a manner similar to that previously described, for example, by applying a voltage across a programmable element of the dot product engine to set the conductivity of the programmable element.

[0069] Figure 7 is a simplified diagram of a transformation method 700 according to some embodiments. As part of the transformer that processes the LLM, the processor 102 (see Figure 1 ) performs the transformation method 700. For example, the transformation method 700 can be performed as part of a multi-head attention operation.

[0070] Processor 102 performs step 702 of obtaining an input sequence. The input sequence may be received from a previous transformer. For example, processor 102 may receive an input matrix X (previously described for equations (1) to (4) and Figure 2 , Figure 3 and Figure 4 described), where each row of the input matrix X is an element of the input sequence.

[0071] The processor 102 performs step 704 of controlling the attention engine 200 to compute the relationships between the elements of the input sequence. This computation may include performing the attention operation (described previously with respect to equations (1) to (4)) on the input matrix X. The previously described attention engine 200 may be used by the processor 102 to compute the attention matrix A (described previously with respect to equation (1)) of the input sequence. This may be used in the analog domain Figure 2 Attention Engine 200 or Figure 5 The attention engine 200 performs at least some of this computation.

[0072] The processor 102 performs step 706 of generating an output sequence based on the relationship between the elements of the input sequence (e.g., the attention matrix A, see equation (1)). The output sequence includes the input sequence plus additional information from the attention matrix A. The output sequence can be a hidden state sequence fed to a subsequent transformer or used to predict the word-meta of a fragment.

[0073] Embodiments may achieve advantages. Performing matrix multiplication in the analog domain may be faster than performing the matrix multiplication in the digital domain, especially when different programmable crossbar switch arrays and multiplication circuits may be used to perform matrix generation and multiplication operations simultaneously. Performing such operations may have O(n) times the number of operations in the digital domain. 2) (where n is the sequence length), but has complexity of O(1) in the analog domain. The reduced complexity can also allow analog domain computations to be performed with reduced power relative to digital domain computations. Additionally, utilizing CMOS-based multiplication circuits allows matrix multiplications to be performed without having to repeatedly reprogram the programmable crossbar array. Thus, the performance and reliability of the computing system can be improved.

[0074] The foregoing summarizes the features of multiple examples so that those skilled in the art can better understand the various aspects of the present disclosure. After reference to the description, various modifications and combinations of the illustrative examples and other examples will be apparent to those skilled in the art. Therefore, the appended claims are intended to cover any such modifications.

Claims

1. A device comprising: A first dot product circuit, the first dot product circuit comprising a plurality of query dot product engines and a plurality of key dot product engines, each of the query dot product engines storing a value corresponding to a query weight matrix, each of the key dot product engines storing a value corresponding to a key weight matrix, the first dot product circuit being configured to: generating a simulated query matrix by multiplying a simulated input matrix with the query weight matrix stored in the query dot product engine; and generating a simulated key matrix by multiplying the simulated input matrix with the key weight matrix stored in the key dot product engine; as well as A first multiplier circuit is coupled to the first dot product circuit, the first multiplier circuit being configured to compute an analog output matrix by multiplying the analog query matrix by a transpose of the analog key matrix.

2. The apparatus of claim 1, further comprising: A digital-to-analog converter is coupled to the first dot product circuit, the digital-to-analog converter being configured to receive a digital input matrix and convert the digital input matrix into the analog input matrix.

3. The apparatus of claim 1, further comprising: An analog-to-digital converter is coupled to the first multiplier circuit, the analog-to-digital converter being configured to receive the analog output matrix and convert the analog output matrix into a digital output matrix.

4. The apparatus of claim 1, further comprising: A second dot product circuit, the second dot product circuit comprising a plurality of value dot product engines, each of the value dot product engines storing a value corresponding to a value weight matrix, the second dot product circuit being configured to: generating a simulated value matrix by multiplying the simulated input matrix with the value weight matrix stored in the value dot product engine; as well as A second multiplier circuit is coupled to the second dot product circuit, and the second multiplier circuit is configured to calculate an analog attention matrix by multiplying an analog SoftMax matrix with the analog value matrix.

5. The apparatus of claim 4, further comprising: a digital circuit configured to calculate a digital SoftMax matrix based on the analog output matrix; as well as A digital-to-analog converter is coupled to the digital circuit and the second multiplier circuit, the digital-to-analog converter being configured to receive the digital SoftMax matrix and convert the digital SoftMax matrix into the analog SoftMax matrix.

6. The apparatus of claim 4, further comprising: An analog-to-digital converter is coupled to the second multiplier circuit, the analog-to-digital converter being configured to receive the analog attention matrix and convert the analog attention matrix into a digital attention matrix.

7. The device of claim 1, wherein: The first dot product circuit includes a plurality of programmable crossbar arrays, and each of the programmable crossbar arrays includes one of the query dot product engines and one of the key dot product engines.

8. The device of claim 7, wherein: The programmable crossbar switch array is a memristor array.

9. A device comprising: Digital to analog converters; a first programmable crossbar switch array, the first programmable crossbar switch array comprising first input electrodes and first output electrodes, the first input electrodes being coupled to a first subset of the outputs of the digital-to-analog converters; a second programmable crossbar switch array, the second programmable crossbar switch array comprising second input electrodes and second output electrodes, the second input electrodes being coupled to a second subset of the outputs of the digital-to-analog converter; as well as A current multiplier is coupled to the first output electrode and the second output electrode.

10. The device of claim 9, wherein: Different subsets of the current multipliers are coupled to different subsets of the first output electrodes and to different subsets of the second output electrodes.

11. The apparatus of claim 9, further comprising: a current summer coupled to the current multiplier; a current-to-voltage converter coupled to the current summer; as well as An analog-to-digital converter is coupled to the current-to-voltage converter.

12. The device of claim 11, wherein: The current-to-voltage converter is a transimpedance amplifier and the current multiplier is a Gilbert cell.

13. The apparatus of claim 9, wherein: The first programmable crossbar array further includes a first programmable element at a first intersection of the first input electrode and the first output electrode, and the second programmable crossbar array further includes a second programmable element at a second intersection of the second input electrode and the second output electrode.

14. The device of claim 13, wherein: The first programmable element and the second programmable element are memristors.

15. A method comprising: obtaining an attention engine including a first dot product circuit and a multiplier circuit coupled to the first dot product circuit, the first dot product circuit including a plurality of query dot product engines and a plurality of key dot product engines; Analyze learning data; as well as A query dot product engine and a key dot product engine of the attention engine are programmed based on an analysis of the learning data.

16. The method of claim 15, wherein: The query dot product engine and the key dot product engine each include a memristor array.

17. The method of claim 15, wherein: The multiplier circuit includes a current multiplier.

18. The method of claim 15, wherein: Programming the query dot product engine and the key dot product engine includes: Calculating a query weight matrix and a key weight matrix; and The query weight matrix and the key weight matrix are stored in the query dot product engine and the key dot product engine, respectively.

19. The method of claim 18, wherein: The query dot product engine and the key dot product engine each include a programmable element, and storing the query weight matrix and the key weight matrix includes: A voltage is applied across the programmable element.

20. The method of claim 15, wherein: The attention engine further comprises a second dot product circuit, the second dot product circuit comprising a plurality of value dot product engines, and the method further comprises: A value dot product engine of the attention engine is programmed based on an analysis of the learning data.

Images