Customized temporary memory for partial dot product reduction

Abstract

Aspects of the disclosed technology include techniques and mechanisms for partial dot product reduction using customized temporary memory. The custom temporary memory may be a dedicated memory that is dedicated to receive and store the partial dot product determined by the matrix multiplier unit. Each partial dot product may correspond to a tile of a resulting matrix, where the resulting matrix is a product of matrix multiplications that may use a first matrix representing a user query as a left operand, and a second matrix representing a user query as a right operand. And using a second matrix representing a trained model containing data available to respond to the user query as a right side operand. The custom register memory may append tiles determined by matrix multiplication, where the appended tiles may create a resulting matrix. Customized temporary memory may write the resulting matrix to general purpose memory, where the resulting matrix may be used to respond to a user query.

Description

[0001] Cross-references

[0002] This application is a continuation of U.S. Patent Application No. 18 / 218,448, filed July 5, 2023, the disclosure of which is hereby incorporated herein by reference. Background Technology

[0003] Current computing chips within computing devices, such as hardware accelerators, can use matrix multiplication to compute data. Matrix multiplication can be performed using two matrices, such as matrix A and matrix B, where matrix multiplication can be represented as A × B.

[0004] Computing devices can perform matrix multiplication using two main memory levels—general-purpose memory and register files. Register files can have a smaller storage capacity than general-purpose memory, but can be a lower-power and higher-bandwidth storage option. Due to the smaller storage capacity of register files, any data exceeding the capacity of the register files can be written to general-purpose memory. General-purpose memory can have a larger storage capacity than register files, but can have lower bandwidth. To compute matrix C, matrices A and B can be written to general-purpose memory.

[0005] The computing device can read sub-parts of matrices A and B from general-purpose memory to perform matrix multiplication. A first tile can be extracted from matrix A, and a second tile from matrix B. The matmul() function can be used to multiply the first tile by the second tile to produce a partial multiplication tile of matrix C. The computing device can write the resulting tile of matrix C to a register file; or, if the register file has reached its maximum storage capacity, to general-purpose memory. The computing device can read tiles of matrix C from general-purpose memory and can use element-wise addition to combine the determined resulting tiles to generate a portion of matrix C, then write that portion back to general-purpose memory. The generated portions of matrix C can be repeatedly read from general-purpose memory, appended to newly generated tiles of matrix C, and written back to general-purpose memory until the entire matrix C is generated.

[0006] The computing device can repeatedly execute the matmul() function until the entire matrix A is multiplied by the entire matrix B to produce the entire matrix C. Each repetition of the matrix multiplication may involve reading two blocks from general-purpose memory and writing the resulting partial multiplication block of matrix C back to general-purpose memory. Depending on the sizes of matrices A and B, repeatedly reading data from and writing data to general-purpose memory can cause processing latency and may consume significant amounts of power and bandwidth.

[0007] Moreover, as computing capabilities of computing devices increase, the demand for memory can also increase, but current memory storage capacities can not grow as quickly as computing capabilities increase. This problem is further exacerbated by general-purpose memory storing additional data that the computing device can run and execute other user applications and programs can require. Thus, constantly reading data from and writing data to the general-purpose memory, as well as sharing the general-purpose memory capacity with data for other operations, can impact user-facing front-end applications that rely on matrix multiplication to generate responses to user queries and provide the responses to the user. That is, a delay in performing matrix multiplication to generate a resulting matrix C can result in a delay in responding to a user query. SUMMARY

[0008] Aspects of the disclosed technology include methods, devices, systems, and computer-readable media associated with using a custom scratch memory for partial dot product reductions. The custom scratch memory can be a dedicated memory that is specifically used to receive and store partial dot products determined by a matrix multiplier unit. Each partial dot product can correspond to a block of a resulting matrix. The resulting matrix can be a product of a matrix multiplication that uses a first matrix as a left-hand operand and a second matrix as a right-hand operand. Execution of the matrix multiplication can generate one or more blocks of the resulting matrix. The custom scratch memory can append the blocks determined by the matrix multiplication, and the appended blocks can create the resulting matrix. The custom scratch memory can write the resulting matrix to a general-purpose memory. The resulting matrix can be used to complete a computation in one or more data computations. By using the custom scratch memory, the speed and efficiency of the data computations can be improved. In some instances, a processing element can perform read and write transactions to the custom scratch memory. The computing device can read the resulting matrix from the general-purpose memory.

[0009] The use of custom scratch memory can reduce the number of read and write transactions performed on general-purpose memory, thereby reducing the risk of processing bottlenecks. For example, partial product tiles resulting from multiplying a first matrix and a second matrix can be stored in custom scratch memory. Repeated read and write transactions can be performed on the custom scratch memory to read the stored partial product tiles and to write subsequently determined partial product tiles to the custom scratch memory, thereby freeing general-purpose memory to perform other applications. Moreover, since the first matrix and the second matrix can be stored in the custom scratch memory along with the extracted arrays from the first matrix and the second matrix, determining partial dot product reductions can not rely on read and write transactions performed on general-purpose memory, thereby increasing processing speed and thus reducing response time. Since general-purpose memory can be used to store data and instructions needed to perform other applications on the computing device, access to the general-purpose memory for performing matrix multiplication and determining partial dot product results can be limited. Thus, the custom scratch memory provides a specialized computing environment that can be smaller and faster than general-purpose memory and can allow efficient performance of matrix multiplication and partial dot products that can be used, for example, to generate responses to user queries.

[0010] One aspect of the present disclosure provides a system for determining partial dot product reductions of data computations, the system comprising: a specialized memory; one or more computing devices configured to communicate with the specialized memory and a general-purpose memory; and instructions that, when executed, cause the one or more computing devices to: receive, from a matrix multiplier unit configured to perform matrix multiplication corresponding to the one or more data computations and associated with the one or more computing devices, a partial dot product between a first tile and a second tile, wherein the first tile is extracted from a first matrix and the second tile is extracted from a second matrix; store the received partial dot product; append the received partial dot product to previously determined partial dot products stored in the specialized memory; generate a resulting matrix based on the appending; write the resulting matrix to the general-purpose memory; and compute the one or more data computations using the resulting matrix. The use of custom scratch memory can improve the efficiency and processing speed of determining the resulting matrix, thereby improving the speed and efficiency of completing the data computations. Moreover, implementing custom scratch memory will generally require less energy than writing to general-purpose memory.

[0011] According to some examples, the first tile includes column elements extracted from the first matrix. According to some examples, the second tile includes row elements extracted from the second matrix. According to some examples, the instructions, when executed, further cause the one or more computing devices to store the first tile and the second tile.

[0012] According to some examples, the instructions, when executed, further cause the one or more computing devices to store an updated first tile, wherein the updated first tile includes new column elements extracted from the first matrix, and wherein the new column elements of the updated first tile are different from the column elements of the first tile.

[0013] According to some examples, the instructions, when executed, further cause the one or more computing devices to store an updated second tile, wherein the updated second tile includes new row elements extracted from the second matrix, and wherein the new row elements of the updated second tile are different from the row elements of the second tile.

[0014] According to some examples, the instructions, when executed, further cause the one or more computing devices to receive, from the matrix multiplier unit, subsequent partial dot products between: the updated first tile and the second tile; and the first tile and the updated second tile.

[0015] In the foregoing embodiment, the instructions, when executed, further cause the one or more computing devices to store the subsequent partial dot products received from the matrix multiplier unit.

[0016] In some examples, each of the partial dot product and the subsequent partial dot product corresponds to a tile of the resulting matrix.

[0017] In the foregoing embodiment, the appending further causes the one or more computing devices to: locate, within the dedicated memory, tiles of the resulting matrix, wherein the tiles are based on the partial dot product and the subsequent partial dot product; and accumulate, using an adder, the located tiles to create a partial resulting matrix until a sum of the located tiles generates the resulting matrix.

[0018] In the foregoing embodiment, the instructions, when executed, further cause the one or more computing devices to: store the partial resulting matrix; and append, using the adder, additional tiles of the resulting matrix to the stored partial resulting matrix to generate an updated partial resulting matrix until the resulting matrix is generated.

[0019] In the foregoing embodiment, the instructions, when executed, further cause the one or more computing devices to append, using the adder, the received partial dot product to previously determined partial dot products until a sum of the received partial dot products indicates that an entirety of the first matrix is multiplied by an entirety of the second matrix.

[0020] Another aspect of this disclosure provides a method for determining partial dot product reduction of data computation using a dedicated memory, the method comprising: receiving, by the dedicated memory, a partial dot product between a first block and a second block from a matrix multiplier unit configured to perform matrix multiplications corresponding to one or more data computations, wherein the first block is extracted from a first matrix and the second block is extracted from a second matrix; storing the received partial dot product in the dedicated memory; appending the received partial dot product to a previously determined partial dot product stored in the dedicated memory by the dedicated memory; generating a resulting matrix by a computing device including the dedicated memory based on the append; writing the resulting matrix into a general-purpose memory by the computing device; and using the resulting matrix to compute the one or more data computations.

[0021] In the foregoing embodiments, the method further includes receiving a subsequent partial dot product between the following items from the matrix multiplier unit by the dedicated memory: an updated first block and the second block, wherein the updated first block includes new column elements extracted from the first matrix, and wherein the new column elements of the updated first block are different from the column elements of the first block; and the first block and the updated second block, wherein the updated second block includes new row elements extracted from the second matrix, and wherein the new row elements of the updated second block are different from the row elements of the second block.

[0022] In some examples, each of the partial dot product and the subsequent partial dot product corresponds to a block of the resulting matrix.

[0023] In some examples, the addition further includes: locating blocks of the resulting matrix by the dedicated memory, wherein the blocks are based on the partial dot product and the subsequent partial dot product; and using an adder by the dedicated memory to accumulate the located blocks to create a partial resulting matrix, until the sum of the located blocks generates the resulting matrix.

[0024] In the foregoing embodiments, the method further includes: storing the partially obtained matrix in the dedicated memory; and using the adder in the dedicated memory to append additional blocks of the obtained matrix to the stored partially obtained matrix to generate an updated partially obtained matrix until the obtained matrix is ​​generated.

[0025] Another aspect of this disclosure provides a non-transitory computer-readable storage medium storing instructions that, when executed by a computing device including one or more processors, dedicated memory, and general-purpose memory, cause the computing device to: receive a partial dot product between a first block and a second block from a matrix multiplier unit configured to perform matrix multiplications corresponding to one or more data computations, wherein the first block is extracted from a first matrix and the second block is extracted from a second matrix; store the received partial dot product in the dedicated memory; append the received partial dot product to a previously determined partial dot product stored in the dedicated memory; generate a resulting matrix based on the append; write the resulting matrix to the general-purpose memory; and use the resulting matrix to determine the one or more data computations.

[0026] In some examples, when the instruction is executed, it further causes the computing device to store the first block and the second block in the dedicated memory.

[0027] In some examples, when the instruction is executed, it further causes the computing device to receive a subsequent partial dot product from the matrix multiplier unit between: an updated first block and the second block, wherein the updated first block includes new column elements extracted from the first matrix, and wherein the new column elements of the updated first block are different from the column elements of the first block; and the first block and the updated second block, wherein the updated second block includes new row elements extracted from the second matrix, and wherein the new row elements of the updated second block are different from the row elements of the second block. Attached Figure Description

[0028] Figure 1 Example matrix multiplications that can be partially reduced using a custom temporary memory are shown according to various aspects of this disclosure.

[0029] Figure 2 A block diagram is shown illustrating an example environment in which a custom temporary storage can be used to determine a partial dot product reduction according to various aspects of this disclosure.

[0030] Figure 3 A flowchart illustrating an example method for partial dot product reduction using a custom temporary memory according to various aspects of this disclosure is shown.

[0031] Figure 4 An example addition is shown, representing a partial result of the matrix obtained according to various aspects of this disclosure.

[0032] Figure 5 An example process or method for partial dot product reduction using a custom temporary storage memory is shown according to various aspects of this disclosure. Detailed Implementation

[0033] This technique relates to partial dot product reduction using a custom scratchpad memory. The technique described herein relates to a custom scratchpad memory for partial dot product reduction. The custom scratchpad memory can be a smaller, dedicated memory used to store partial results of matrix multiplication, such as a partial multiplication block of the resulting matrix C generated in each iteration of the multiplication of matrices A and B, denoted as A×B. In some instances, the custom scratchpad memory can be referred to as a partial result buffer, which is used to store partial results of matrix multiplication. A custom scratchpad memory can be used instead of storing partial results of matrix multiplication in general-purpose memory. This reduces the number of read transactions from and write transactions to general-purpose memory until the entire matrix C has been generated and can be stored in general-purpose memory.

[0034] Figure 1 Example matrix multiplication operations that can be used to determine partial dot product reduction according to aspects of this disclosure are illustrated. The partial dot product can be determined based on matrix multiplication performed on matrices A and B, denoted as... For example, a computing device can perform matrix multiplication 100 using matrices A 110 and B 120. Matrix A 110 can be a left-hand side (LHS) operand and can represent user input, such as a search query, an image, input text, etc. In some examples, matrix A can include inputs to neural network layers or other parts of a machine learning model; outputs from one neural network layer to a subsequent neural network layer; learned embeddings mapped to inputs of a machine learning model; text or other data encoded as a numerical representation, such as integers, floating-point values, or other numerical formats with different levels of precision; pixel values ​​of an image or video frame; feature values ​​of one or more inputs to a machine learning model, etc. Matrix B 120 can be a right-hand side (RHS) operand and can represent a trained model or a portion of a trained model, such as weights or biases of a neural network layer.

[0035] For example, the data in matrix A 110 may include user queries submitted to, for example, an automated search engine or automated chatbot using a large language model, which may be represented by matrix B 120. Matrix multiplication 100 can be used to run the user query via the search engine or chatbot to generate a response to the user query, where the response is based on the information available in matrix B 120. The output of the matrix multiplication of matrices A 110 and B 120 may be the resulting matrix C 130, which may contain the response to the user query, and this response may be transmitted back to the user or to another neural network layer.

[0036] Matrix B 120 can be loaded into one or more systolic arrays. The sizes of matrices A 110 and B 120 can depend on the application, but the size and shape of multiple systolic arrays used for matrix multiplication 100—such as the array used for matrix B 120—can be fixed. Matrix B 120 can be buffered within an array, for example, a systolic array, and matrix A 110 can be streamed through matrix B 120 to produce matrix C 130, which can be represented as... The following diagram discusses matrix multiplication 100 in more detail.

[0037] Figure 2 A block diagram is shown of an example environment in which a custom temporary storage memory can be used to determine a partial dot product reduction according to aspects of this disclosure. Matrix multiplication 100 can be implemented using one or more devices having one or more processors in one or more locations—such as server computing device 210 and user computing device 220 in computing environment 200. Server computing device 210 and user computing device 220 can be communicatively coupled to one or more storage devices via a network. The storage devices can be a combination of volatile and non-volatile memory and can be located in the same or different physical locations as the computing devices. For example, the storage devices can include any type of non-transitory computer-readable medium capable of storing information, such as hard disk drives, solid-state drives, magnetic tape drives, optical storage, memory cards, ROM, RAM, DVDs, CD-ROMs, writable memory, and read-only memory. In some instances, database 250 can store data transferred across network 230, as well as data transferred between server computing device 210, user computing device 220, and data center 240.

[0038] Server computing device 210 may include one or more processors and memories, such as processor 201 and memory 202, which are referred to herein as processor 201 and memory 202, respectively. Memory 202 may include custom scratch memory 205 and may correspond to additional memory locations associated with server computing device 210 as discussed below. Memory 202 may also include instruction 203 and data 204. Server computing device 210 may further include compiler 206, matrix multiplier unit (MXU) 207, gain matrix register (GMR) 208, and matrix scratch register (MSR) 209. In some instances, MXU 207, GMR 208, and MSR 209 may be part of memory 202.

[0039] Memory 202 may store information accessible by processor 201, including instructions 203 executable by processor 201. Memory 202 may also include data 204 that can be retrieved, manipulated, or stored by processor 201. Memory 202 may be a type of non-transitory computer-readable medium capable of storing information accessible by processor 201, such as volatile memory and non-volatile memory. Processor 201 may include one or more central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0040] In some instances, memory 202 may correspond to an additional memory location associated with server computing device 210, such as high-bandwidth memory (HBM) or virtual memory (Vmem). Server computing device 210 may read data (such as matrices) from either HBM or Vmem to determine the dot product reduction discussed below. For example, server computing device 210 may read a matrix from Vmem that can be used to perform matrix multiplication.

[0041] In some instances, memory 202 may store instructions 203 and data 204 corresponding to additional operations, functions, and / or programs performed by server computing device 210. Therefore, memory 202 may be referred to as general-purpose memory. As general-purpose memory, the demand for memory 202 may increase as server computing device 210 performs other functions or operations besides determining partial dot product reduction.

[0042] Therefore, memory 202 can be dedicated to determining partial dot product reduction in custom scratch memory 205. Custom scratch memory 205 can be a smaller dedicated memory that can store the matrix used to perform matrix multiplication and determine partial dot product reduction. In some instances, server computing device 210 can read data or collect resources needed to determine partial dot product reduction, such as matrices and multiple chunks extracted from the matrices, from at least memory 202 or data center 240. Server computing device 210 can write multiple chunks extracted from the matrix into custom scratch memory 205, such that the determination of partial dot product reduction is performed within custom scratch memory 205. Server computing device 210 can determine the resulting matrix based on matrix multiplication and write the resulting matrix into memory 202, where server computing device 210 can use the resulting matrix to generate a response to a user query.

[0043] Data in the custom temporary storage 205 persists during the matrix multiplications required to generate the resulting matrix. After the resulting matrix is ​​generated, the server computing device 210 can erase the custom temporary storage 205, allowing it to receive and store new data needed to perform subsequent matrix multiplication operations to generate responses to subsequently received user queries.

[0044] Instruction 203 may include one or more instructions that, when executed by processor 201, cause processor 201 to perform the action defined by instruction 203. Instruction 203 may be stored in object code format for direct processing by processor 201, or stored in other formats, including interpreted scripts or a collection of independent source code modules that are interpreted or pre-compiled as needed. Figure 1 The matrix multiplication 100 determines the instructions for partial dot product reduction. The matrix multiplication can be performed using processor 201 and / or other processors located remotely from server computing device 210.

[0045] Processor 201 can retrieve, store, or modify data 204 according to instructions. Data 204 can be stored in computer registers, stored as a table with multiple different fields and records in a relational or non-relational database, or stored as a JSON, YAML, proto, or XML document. Data 204 can also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data 204 may include information sufficient to identify relevant information such as numbers, descriptive text, proprietary codes, pointers, or references to data stored in other memory locations, including other network locations.

[0046] Server computing device 210 can use compiler 206, MXU 207, GMR 208, and MSR 209 to perform the matrix multiplications required to determine partial dot product reduction. Server computing device 210 can use compiler 206 to read multiple blocks extracted from the matrix from memory 202 and write these blocks to MXU 207. MXU 207 can stream a first block through a second block, where the first block can be extracted from matrix A, and the second block can be extracted from matrix B. MXU 207 can store the second block and can multiply the first block with the second block. At a future point in time, MXU 207 may complete the multiplication of the first block with the second block and may need to extract an updated second block from matrix B. The updated second block extracted from matrix B may differ from the block previously extracted from matrix B.

[0047] The MXU 207 can use a double buffer to store the updated second block extracted from matrix B, which is also used to perform matrix multiplication. The double buffer can store the block currently used for matrix multiplication, as well as the new block to be used for matrix multiplication. Specifically, the MXU 207 can read the currently used block from GMR 208 and write a new block to MSR 209. When the matrix multiplication using the block read from GMR 208 is complete, the MXU 207 can read the new block from MSR 209 and write it to GMR 208. The MXU 207 can continue this process until the entire matrix multiplication is complete.

[0048] User computing device 220 may also be configured similarly to server computing device 210, having one or more processors 221, memory 222, instructions 223, and data 224. User computing device 220 may also include user input 225 and user output 226. User input 225 may include any suitable mechanisms or technologies for generating user input, such as a keyboard, mouse, mechanical actuator, soft actuator, touchscreen, microphone, and sensors. User output 226 may additionally include one or more speakers, transducers or other audio outputs, haptic interfaces, or other tactile feedback providing non-visual and non-audible information to the platform user of user computing device 220.

[0049] User computing device 220 can be configured to transmit data to server computing device 210, and server computing device 210 can be configured to display at least a portion of the received data on a display.

[0050] although Figure 2 While processors and memory are shown as being within a computing device, the components described herein may include multiple processors and memories that can operate in different physical locations rather than within the same computing device. For example, some of the instructions and data may be stored on a removable SD card, while others may be stored within a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from the processor but still accessible to the processor. Similarly, processors may include a collection of processors capable of performing concurrent and / or sequential operations. The computing device may each include one or more internal clocks that provide timing information, which can be used for time measurement of operations and programs run by the computing device.

[0051] Server computing device 210 can be connected via network 230 to data center 240, which houses any number of hardware accelerators. Data center 240 may be one of multiple data centers or other facilities housing various types of computing devices such as hardware accelerators. Computing resources housed within the data center may be specified to use custom scratch memory to determine partial dot product reduction, as described herein.

[0052] Data center 240 may include multiple hardware accelerators, such as hardware accelerators 260A-N. Hardware accelerators 260A-N may be any type of processor, such as a GPU, FPGA, or ASIC such as a TPU. Aspects of this disclosure may be implemented as dedicated functions of a general-purpose processor in some examples, for example, as part of a CPU or another processor configured to perform dot product reduction using a custom scratch memory as described herein.

[0053] Server computing device 210, user computing device 220, and data center 240 can communicate directly and indirectly via a network. For example, using network sockets, user computing device 220 can connect to a service operating on server computing device 210 via Internet Protocol. In some instances, user computing device 220 can connect to a service operating within data center 240. The devices and data center 240 can configure listening sockets that can accept initiating connections for sending and receiving information—such as user queries and responses to user queries. The network itself can include various configurations and protocols, including the Internet, World Wide Web, intranet, virtual private network, wide area network, local area network, and private networks using one or more company-proprietary communication protocols. The network can support various short-range and long-range connections. Short-range and long-range connections can be made on different bandwidths, such as 2.402 GHz to 2.480 GHz, typically associated with the Bluetooth® standard, and 2.4 GHz and 5 GHz, typically associated with the Wi-Fi® communication protocol; or using multiple communication standards, such as the LTE® standard for wireless broadband communication. The network can also support wired connections between devices and data centers, including over various types of Ethernet connections.

[0054] Although Figure 2 The disclosure illustrates a single server computing device, a user computing device, and a data center; however, it should be understood that aspects of this disclosure can be implemented according to various configurations and magnitudes of computing devices (including in paradigms for sequential or parallel processing), or through a distributed network of multiple devices. In some implementations, aspects of this disclosure can be performed on a single device connected to hardware accelerators or any combination thereof configured to optimize partial dot product reduction determination.

[0055] Server computing device 210 can be configured to receive user requests from user computing device 220, where the user requests may relate to various topics. Server computing device 210 can also be configured to receive requests for processing data from user computing device 220 using computing resources in data center 240. For example, computing environment 200 may be part of a computing platform configured to provide various services to users through various user interfaces and / or application programming interfaces (APIs) exposing platform services. These various services may include responding to received user queries, where the response is determined using partial dot product reduction with a custom temporary storage. User computing device 220 may transmit the user query to server computing device 210. Server computing device 210 may generate a first matrix, where the first matrix may represent the user query. Server computing device 210 may perform matrix multiplication using a portion of the first matrix and a portion of a second matrix. The second matrix may represent model parameter values ​​of a trained model or a model being trained, which can be used to generate a response to the user query. Server computing device 210 may multiply the portion of the first matrix with a portion of the second matrix to generate a partial result of the resulting matrix C. The server computing device 210 can store this partial result in a custom temporary storage memory until the matrix multiplication produces the entire resulting matrix C. In some examples, the multiplied matrices are tensors or parts of tensors.

[0056] In some instances, server computing device 210 may read the second matrix from resources stored within data center 240—such as disks or volatile memory. Additionally, in some examples, server computing device 210 may read the second matrix from memory 202.

[0057] As another example of potential services provided by the platform implementing computing environment 200, server computing device 210 can maintain multiple models for determining partial dot product reductions using custom scratch memory, depending on the different constraints available within data center 240. For example, server computing device 210 can maintain different families of models for deployment on various types of TPUs and / or GPUs housed in data center 240 or otherwise available for processing.

[0058] Figure 3A flowchart illustrating an example method for partial dot product reduction using a custom temporary storage memory according to various aspects of this disclosure is shown. The operations described herein are presented in the current order as an example, and this order is not intended to be limiting. Furthermore, operations may be omitted from or added to the example method. The methods described herein can be performed by the devices and components shown in example environment 200—such as server computing device 210, user computing device 220, and data center 240. In some examples, the methods described herein may be performed using additional or alternative computing devices and components therein.

[0059] At box 301, server computing device 210 can receive user queries from user computing device 220. User queries may include search queries, images, input text, etc. User queries may include requests for specific information, such as search query results, image tags, responses to input text, etc. Server computing device 210 can generate a response to the received user query by performing matrix multiplication using a first matrix corresponding to the received user query and a second matrix corresponding to a trained model or a trained model including information that can be used to respond to the user query.

[0060] At box 302, server computing device 210 can convert the received user query into a matrix representation of the user query, referred to herein as matrix A. Server computing device 210 can write matrix A into memory 202.

[0061] At box 303, server computing device 210 may read a second matrix, referred to herein as matrix B, from memory 202. In an instance where matrix B is written to data center 240, server computing device 210 may read matrix B from data center 240 and write matrix B to memory 202. In some instances, server computing device 210 may read matrix B from high-bandwidth memory (HBM) therein and may write matrix B to virtual memory (Vmem) therein.

[0062] At box 304, server computing device 210 can read portions of matrices A and B from memory 202, such as blocks of matrices A and B, where a block of matrix A may include elements of matrix A, and a block of matrix B may include elements of matrix B. Specifically, server computing device 210 can extract a first block from matrix A and a second block from matrix B. The first and second blocks can be used to perform matrix multiplication of matrices A and B. In some instances, the first block may include elements of matrix A, such as a first subset of the column elements of matrix A. In some instances, the second block may include elements of matrix B, such as a first subset of the row elements of matrix B. Server computing device 210 can write the first and second blocks to custom temporary storage 205.

[0063] At box 305, server computing device 210 can perform matrix multiplication using a first block extracted from matrix A and a second block extracted from matrix B. Server computing device 210 can perform matrix multiplication using at least one array function—such as the matmul() function—configured to determine the matrix product of two arrays or submatrices—such as blocks. The matmul() function can be used to determine a partial multiplication block of the resulting matrix C, and may require two blocks to compute the matrix product. To perform matrix multiplication, the height of the first block, such as a column block extracted from matrix A, should match the width of the second block, such as a row block extracted from matrix B. For example, a first block with a height of 128 units can be multiplied by a second block with a width of 128 units. The product of this multiplication can be a partial result of matrix C, such as a 128x128 block of matrix C.

[0064] At box 306, server computing device 210 can write a portion of the result of matrix C into custom temporary storage 205. For example, server computing device 210 can write a determined block of the obtained matrix C into custom temporary storage 205.

[0065] At box 307, server computing device 210 can determine whether the entire resulting matrix C was generated according to the matrix multiplication. To do this, server computing device 210 can determine whether each block of matrix A is multiplied by each block of matrix B. In some instances, server computing device 210 can determine whether each column block of matrix A is multiplied by each row block of matrix B.

[0066] If, at box 307, the server computing device 210 determines that the entire resulting matrix C has not yet been generated, then at box 308, the server computing device 210 can append a portion of the result of matrix C to the previously determined portion of the resulting matrix C. Specifically, the server computing device 210 can append a determined block of the resulting matrix C to a previously determined block of the resulting matrix C. To do this, the server computing device 210 can read the previously determined block of the resulting matrix C from the custom temporary storage 205, and can use, for example, an adder to append the determined block of the resulting matrix C to the previously determined block.

[0067] The server computing device 210 can use an adder to generate fully filled columns or rows of the obtained matrix C based on determined blocks of the obtained matrix C. The adder can have read-accumulate-write capabilities. Therefore, the adder can be configured to read determined blocks of the obtained matrix C and previously determined blocks of the obtained matrix C, accumulate the sum of the determined blocks of the obtained matrix C to generate a partial result of the obtained matrix C, and write the partial result of the obtained matrix C to a custom temporary storage 205.

[0068] Server computing device 210 may return to box 304 and repeat the process described herein until each block of matrix A is multiplied by each block of matrix B to generate each partial multiplication block of the resulting matrix C. To do this, server computing device 210 may extract an updated first block from matrix A, wherein the updated first block may include new elements extracted from matrix A, such as column elements. The updated first block may include elements different from those elements in the first block, such as different column elements. Server computing device 210 may use the updated first block and second block to repeat matrix multiplication. Performing matrix multiplication using the updated first block and second block produces additional blocks of the resulting matrix C. Server computing device 210 may append the additional blocks of the resulting matrix C to the partial result of the resulting matrix C and may determine whether the entire resulting matrix C has been generated. If the entire resulting matrix C has not yet been generated, server computing device 210 may return to box 304. Server computing device 210 may repeat the process described herein until each block of matrix A is multiplied by a block within matrix B.

[0069] In some instances, server computing device 210 may return to box 304 and may repeat the process described herein until each block of matrix B—e.g., each row block extracted from matrix B—is multiplied by each block of matrix A to generate the entirety of the resulting matrix C. To do this, server computing device 210 may extract an updated second block from matrix B, wherein the updated second block may include new elements extracted from matrix B, such as row elements, or new row elements. The updated second block may include elements that differ from those in the second block, such as different row elements. Server computing device 210 may use the first block and the updated second block to repeat matrix multiplication. Performing matrix multiplication using the first block and the updated second block produces additional blocks of the resulting matrix C. Server computing device 210 may append the additional blocks of the resulting matrix C to the partial result of the resulting matrix C and may determine whether the entirety of the resulting matrix C has been generated. If the entirety of the resulting matrix C has not yet been generated, server computing device 210 may return to box 304. The server computing device 210 may repeat the process described herein until the first block is multiplied by each block extracted from matrix B—such as each row block.

[0070] Server computing device 210 can use MXU 207 and a double buffer to store an updated second block, which is used to perform matrix multiplication. The double buffer can store the two blocks currently used for matrix multiplication, as well as a new block to be used for matrix multiplication. Specifically, MXU 207 can read the currently used block from GMR 208 and write a new block to MSR 209. When matrix multiplication using the block read from GMR 208 is complete, MXU 207 can read a new block from MSR 209 and write it to GMR 208. MXU 207 can continue this process until the entire matrix multiplication is complete. After the matrix multiplication is complete, server computing device 210 can determine whether the entire resulting matrix C has been generated. If the entire resulting matrix C has not yet been generated, server computing device 210 can return to box 304.

[0071] Figure 4 An example addition is shown, representing a partial result of the matrix obtained according to various aspects of this disclosure. Figure 4 The example addition shown is characterized by the iteration of the matrix multiplication described in this paper. This paper will combine... Figure 3 Example method discussion Figure 4 Example addition of partial results of the resulting matrix shown.

[0072] The first iteration 402 of the matrix multiplication illustrates the matrix multiplication of matrices A and B, where the first block comprises a column block containing elements 1 and 2 of matrix A, and the second block comprises a row block containing elements 1 and 2 of matrix B. The matrix multiplication of this first and second block can produce a partial multiplication block containing elements 1 and 2 of the resulting matrix C. In some instances, the first iteration 402 of the matrix multiplication can be combined with... Figure 3 The example method shown corresponds to the initial iteration. Based on the determination that the resulting matrix C has not yet been generated, the server computing device 210 can repeat the matrix multiplication using the updated blocks.

[0073] Repeated matrix multiplication using the updated blocks can correspond to the second iteration 404 of matrix multiplication, which illustrates the matrix multiplication of matrices A and B, where the updated first block includes a column block containing elements 3 and 4 of matrix A, and the updated second block includes a row block containing elements 3 and 4 of matrix B. Matrix multiplication of the updated first and second blocks can produce a partial multiplication block containing elements 3 and 4 of the resulting matrix C. The first iteration 402 and the second iteration 404 of matrix multiplication can be appended using adder 408 to produce an updated resulting matrix C. This updated resulting matrix C can include a partial multiplication block containing elements 1 through 4 of the first column of the resulting matrix C. In some instances, the second iteration 404 of matrix multiplication can be compared with... Figure 3 The second iteration of the example method shown corresponds to this. Based on the determination that the resulting matrix C has not yet been generated, the server computing device 210 can use the updated blocks to repeat the matrix multiplication described above.

[0074] Repeated matrix multiplication using the updated blocks can correspond to the third iteration 406 of matrix multiplication, which illustrates the matrix multiplication of matrices A and B, where the updated first block includes a column block containing elements 5 and 6 of matrix A, and the updated second block includes a row block containing elements 5 and 6 of matrix B. Matrix multiplication of the updated first and second blocks can produce a partial multiplication block containing elements 5 and 6 of the resulting matrix C. The second iteration 404 and the third iteration 406 of matrix multiplication can be appended using adder 408 to produce an updated resulting matrix C. This updated resulting matrix C can include a partial multiplication block containing elements 1 through 6 of the first column of the resulting matrix C. In some instances, the third iteration 406 of matrix multiplication can be compared with... Figure 3 The example method shown corresponds to the third iteration.

[0075] The server computing device 210 can be repeated Figure 4 The example shown is partially added until the resulting matrix C is generated and written to memory 202.

[0076] Return to Figure 3 In box 307, if the server computing device 210 determines that the entirety of the resulting matrix C has been generated, then in box 309, the server computing device 210 can read the entirety of the resulting matrix C from the custom temporary storage 205 and write the entirety of the resulting matrix C into the storage 202.

[0077] At box 310, server computing device 210 can use the resulting matrix C to generate a response to a user query in response to matrix A, and can transmit the generated response to user computing device 220. For example, if matrix A includes a user query submitted to an automated chatbot, the resulting matrix C can be used to respond to a user query after processing the input data of the user query through a trained machine learning model, which may include performing the matrix multiplication described herein. The contents of the resulting matrix C can be used to update model parameter values, such as parameter values ​​that form at least a portion of matrix B multiplied by matrix A. The updated model can be used to respond to subsequent user queries.

[0078] After the matrix multiplication is terminated, the custom temporary storage 205 can be cleared, and the process can be repeated when subsequent user input is received.

[0079] Figure 5 Example procedures or methods for partial dot product reduction using a custom scratchpad memory are illustrated according to various aspects of this disclosure. The operations described herein are presented in the current order as an example, and this order is not intended to be limiting. Furthermore, operations may be omitted from or added to the example methods. The methods described herein can be performed by the apparatus and components shown in example environment 200—such as custom scratchpad memory 205. In some examples, the methods described herein can be performed using additional or alternative computing devices and components therein.

[0080] At block 501, custom scratchpad memory 205 can receive a partial dot product between a first block and a second block from MXU 207, wherein the first block may contain column elements extracted from a first matrix, and the second block may contain row elements extracted from a second matrix. As described above, server computing device 210 can use MXU 207 to perform matrix multiplication and store blocks extracted from the matrix currently used for performing matrix multiplication, as well as updated blocks that will be used for performing matrix multiplication at a future time. In some instances, custom scratchpad memory 205 can store the first block and the second block, and MXU 207 can read blocks from custom scratchpad memory 205 to perform matrix multiplication. Furthermore, in some instances, custom scratchpad memory 205 can store updated first blocks and updated second blocks, wherein the updated first block may include new column elements extracted from the first matrix, and the updated second block may include new row elements extracted from the second matrix.

[0081] MXU 207 can perform matrix multiplication using a first block, a second block, and updated first and second blocks, wherein each iteration of the matrix multiplication can generate a partial dot product between the first and second blocks, between the first and updated second blocks, or between the updated first and second blocks. After the matrix multiplication is complete, MXU 207 can write the determined partial dot product to custom temporary storage 205. In some instances, server computing device 210 can write the determined partial dot product to custom temporary storage 205.

[0082] At box 502, a custom temporary storage memory 205 can store the received partial dot products. When subsequent partial dot products are received from MXU 207, the custom temporary storage memory 205 can continuously store these subsequent partial dot products. The received partial dot products and subsequent received partial dot products can correspond to blocks of the resulting matrix. In some instances, subsequent partial dot products can correspond to matrix multiplication performed using updated first and second blocks, or using the first block and the updated second block.

[0083] At box 503, custom temporary storage 205 can use an adder to append the received partial dot products to previously determined partial dot products already stored in custom temporary storage 205. To do this, custom temporary storage 205 can read previously received partial dot products from memory and accumulate each of the previously received partial dot products to create a partial resulting matrix, such as the resulting matrix C.

[0084] At box 504, custom temporary storage 205 can generate a resulting matrix, such as resulting matrix C, based on this appending. When custom temporary storage 205 receives subsequent partial dot products, it can repeat the process described herein to append the newly received partial dot products to the partial resulting matrix. Appending additional blocks to the partial resulting matrix creates an updated partial resulting matrix, which custom temporary storage 205 can store. Custom temporary storage 205 can continue to append received partial dot products to the updated partial resulting matrix until the entire resulting matrix is ​​generated. The entire resulting matrix can be generated by multiplying the entire first matrix by the entire second matrix.

[0085] At box 505, a custom temporary storage 205 can write the generated matrix to general-purpose memory, such as memory 202. Memory 202 can use the generated matrix to generate a response to a user query.

[0086] The use of custom scratchpad memory 205 reduces the number of read transactions from memory 202 and the number of write transactions to memory 202. This allows server computing device 210 to use memory 202 for additional operations and applications performed on server computing device 210, and reduces the need for server computing device 210 to compete with concurrent operations and applications for memory space within memory 202 to perform matrix multiplication. The use of custom scratchpad memory 205 can offload bandwidth consumed by memory 202 to custom scratchpad memory 205, which can improve the efficiency of matrix multiplication because custom scratchpad memory 205 can be smaller and faster than memory 202 without consuming a large amount of area or power. The use of custom scratchpad memory 205 can reduce the amount of time required to compute the resulting matrix C, and can reduce the amount of time required to respond to user queries based on the resulting matrix C. The use of adders can reduce ALU bandwidth, additional load, and the storage bandwidth that may be required to perform read transactions from memory 202 and write transactions to that memory.

[0087] The aspects of this disclosure can be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, and / or in computer hardware (such as the structures disclosed herein, structural equivalents of said structures, or combinations thereof). The aspects of this disclosure can further be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a tangible, non-transitory computer storage medium, for execution by one or more data processing devices or to control the operation of said one or more data processing devices. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. The computer program instructions can be encoded on artificially generated propagated signals—such as machine-generated electrical, optical, or electromagnetic signals—that are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0088] The term "configured" is used herein in conjunction with both system and computer program components. For a system of one or more computers configured to perform a specific operation or action, it means that the system has software, firmware, hardware, or a combination thereof installed thereon to cause the system to perform that operation or action. For one or more computer programs configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by one or more data processing devices, cause those devices to perform those operations or actions.

[0089] The term "data processing device" refers to data processing hardware and encompasses a variety of devices, apparatuses, and machines used for processing data, including programmable processors, computers, or combinations thereof. Data processing devices may include dedicated logic circuit systems, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Data processing devices may include code that creates an execution environment for computer programs, such as code that constitutes processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0090] Data processing devices may include dedicated hardware accelerator units for implementing machine learning models to handle the general and computationally intensive portions of machine learning training or production, such as inference or workloads. Machine learning models can be implemented and deployed using one or more machine learning frameworks, such as the TensorFlow framework.

[0091] The term "computer program" refers to a program, software, software application, app, module, software module, script, or code. Computer programs can be written in any type of programming language, including compiled, interpreted, declarative, or procedural languages, or combinations thereof. Computer programs can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for a computing environment. Computer programs can correspond to files in a file system and can be stored as part of a file that holds other programs or data (such as one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (such as files storing one or more modules, subroutines, or code portions). Computer programs can execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via data communication networks.

[0092] The term "database" refers to any collection of data. The data can be unstructured or structured in any way. The data can be stored on one or more storage devices in one or more locations. For example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0093] The term "engine" refers to a software-based system, subsystem, or process programmed to perform one or more specific functions. An engine may be implemented as one or more software modules or components or may be installed on one or more computers at one or more locations. A particular engine may have its own dedicated one or more computers, or multiple engines may be installed on and run on the same one or more computers.

[0094] The processes and logical flows described herein can be executed by one or more computers, which execute one or more computer programs to perform functions by manipulating input data and generating output data. The processes and logical flows can also be executed by a dedicated logic circuit system or a combination of a dedicated logic circuit system and one or more computers.

[0095] A computer or special-purpose logic circuit system that executes one or more computer programs may include a central processing unit (CPU) comprising a general-purpose or special-purpose microprocessor for running or executing instructions, and one or more memory devices for storing instructions and data. The CPU may receive instructions and data from one or more memory devices, such as read-only memory, random access memory, or combinations thereof, and may run or execute those instructions. The computer or special-purpose logic circuit system may also include or be operatively coupled to one or more storage devices, such as a magnetic disk, magneto-optical disk, or optical disk, for storing data, to receive data from or transfer data to that storage device. The computer or special-purpose logic circuit system may be embedded in another device, such as, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS), or a portable storage device, such as a universal serial bus (USB) flash drive.

[0096] Computer-readable media suitable for storing one or more computer programs may include any form of volatile or non-volatile memory, medium, or memory device. Examples include: semiconductor memory devices, such as EPROM, EEPROM, or flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; CD-ROM disks; DVD-ROM disks; or combinations thereof.

[0097] The aspects of this disclosure can be implemented in a computing system comprising: a back-end component, such as a data server; a middleware component, such as an application server; or a front-end component, such as a client computer having a graphical user interface, a web browser, or an app, or any combination thereof. The components of the system can be interconnected via any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0098] A computing system may include clients and servers. Clients and servers may be geographically separated and interact via a communication network. The client-server relationship arises from computer programs running on respective computers and having a client-server relationship with each other. For example, a server may transmit data, such as an HTML page, to a client device, for example, for the purpose of displaying data to a user interacting with the client device and receiving user input from that user. Data generated at the client device (e.g., the result of user interaction) may be received from the client device at the server.

[0099] Unless otherwise stated, the foregoing examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description should be presented in an illustrative rather than restrictive manner. Furthermore, the examples described herein and the provision of terms such as “such as,” “comprising,” etc., should not be construed as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of many possible implementations. Moreover, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A system for determining a partial dot product reduction of a data computation, the system comprising: The system comprises: a dedicated memory; a general-purpose memory; and one or more computing devices communicatively coupled to the dedicated memory and general-purpose memory; and instructions for performing one or more data computations that, when executed, cause the one or more computing devices to: receive, from a matrix multiplier unit configured to perform matrix multiplication corresponding to the one or more data computations and associated with the one or more computing devices, a partial dot product between a first block and a second block, wherein the first block is extracted from a first matrix and the second block is extracted from a second matrix; store the received partial dot product; append the received partial dot product to previously determined partial dot products stored in the dedicated memory; generate a resulting matrix based on the appending; write the resulting matrix to the general-purpose memory; and use the resulting matrix to compute the one or more data computations.

2. The system of claim 1, wherein, wherein the first block comprises column elements extracted from the first matrix.

3. The system of claim 1, wherein, wherein the second block comprises row elements extracted from the second matrix.

4. The system of claim 1, wherein, wherein the instructions, when executed, further cause the one or more computing devices to store the first block and the second block.

5. The system of claim 1, wherein, wherein the instructions, when executed, further cause the one or more computing devices to store an updated first block, wherein the updated first block comprises new column elements extracted from the first matrix, and wherein the new column elements of the updated first block are different from the column elements of the first block.

6. The system of claim 1, wherein, wherein the instructions, when executed, further cause the one or more computing devices to store an updated second block, wherein the updated second block comprises new row elements extracted from the second matrix, and wherein the new row elements of the updated second block are different from the row elements of the second block.

7. The system of claim 1, wherein, wherein the instructions, when executed, further cause the one or more computing devices to receive, from the matrix multiplier unit, a subsequent partial dot product between: the updated first block and the second block; and the first block and the updated second block.

8. The system of claim 1, wherein, wherein the instructions, when executed, further cause the one or more computing devices to store the subsequent partial dot product received from the matrix multiplier unit.

9. The system of claim 1, wherein, wherein each of the partial dot product and the subsequent partial dot product corresponds to a block of the resulting matrix.

10. The system of claim 9, wherein, wherein the appending further causes the one or more computing devices to: locate the block of the resulting matrix within the dedicated memory, wherein the block is based on the partial dot product and the subsequent partial dot product; and accumulate the located block using an adder to create a partial resulting matrix until a sum of the located blocks generates the resulting matrix.

11. The system of claim 10, wherein, wherein the instructions, when executed, cause the one or more computing devices to: store the partial resulting matrix; and append additional blocks of the resulting matrix to the stored partial resulting matrix using the adder to generate an updated partial resulting matrix until the resulting matrix is generated. wherein the instructions, when executed, cause the one or more computing devices to: store the partial resulting matrix; and append additional blocks of the resulting matrix to the stored partial resulting matrix using the adder to generate an updated partial resulting matrix until the resulting matrix is generated.

12. The system of claim 1, wherein, wherein the instructions, when executed, further cause the one or more computing devices to append the received partial dot products to previously determined partial dot products using the adder until a sum of the received partial dot products indicates a multiplication of an entirety of the first matrix by an entirety of the second matrix.

13. A method for determining a partial dot product reduction of a data computation using a dedicated memory, characterized in that, The method includes: receiving, by the dedicated memory from a matrix multiplier unit configured to perform a matrix multiplication corresponding to one or more data computations, a partial dot product between a first tile and a second tile, wherein the first tile is extracted from a first matrix and the second tile is extracted from a second matrix; storing, by the dedicated memory, the received partial dot product; appending, by the dedicated memory, the received partial dot product to previously determined partial dot products stored in the dedicated memory; generating, by the computing device comprising the dedicated memory, a resulting matrix based on the appending; writing, by the computing device, the resulting matrix to a general purpose memory; and computing the one or more data computations using the resulting matrix.

14. The method of claim 13, wherein, further comprising receiving, by the dedicated memory from the matrix multiplier unit, a subsequent partial dot product between: an updated first tile and the second tile, wherein the updated first tile comprises new column elements extracted from the first matrix, and wherein the new column elements of the updated first tile are different from column elements of the first tile; and the first tile and an updated second tile, wherein the updated second tile comprises new row elements extracted from the second matrix, and wherein the new row elements of the updated second tile are different from row elements of the second tile.

15. The method of claim 13, wherein, wherein each of the partial dot product and the subsequent partial dot product corresponds to a tile of the resulting matrix.

16. The method of claim 15, wherein, wherein the appending further comprises: locating, by the dedicated memory, the tile of the resulting matrix, wherein the tile is based on the partial dot product and the subsequent partial dot product; and accumulating, by the dedicated memory using an adder, the located tile to create a partial resulting matrix until a sum of the located tiles generates the resulting matrix.

17. The method of claim 16, wherein, further comprising: storing, by the dedicated memory, the partial resulting matrix; and appending, by the dedicated memory using the adder, additional tiles of the resulting matrix to the stored partial resulting matrix to generate an updated partial resulting matrix until the resulting matrix is generated.

18. A non-transitory computer-readable storage medium storing instructions, the method comprising: The instructions, when executed by a computing device comprising one or more processors, a dedicated memory, and a general purpose memory, cause the computing device to: receive, from a matrix multiplier unit configured to perform a matrix multiplication corresponding to one or more data computations, a partial dot product between a first tile and a second tile, wherein the first tile is extracted from a first matrix and the second tile is extracted from a second matrix; store the received partial dot product in the dedicated memory; append the received partial dot product to previously determined partial dot products stored in the dedicated memory; generate a resulting matrix based on the appending; write the resulting matrix to the general purpose memory; and determining the one or more data computations using the resulting matrix.

19. The non-transitory computer-readable storage medium of claim 18, wherein, the instructions, when executed, further cause the computing device to store the first tile and the second tile in the dedicated memory.

20. The non-transitory computer-readable storage medium of claim 18, wherein, the instructions, when executed, further cause the computing device to receive, from the matrix multiplier unit, a subsequent partial dot product between: an updated first tile and the second tile, wherein the updated first tile includes new column elements extracted from the first matrix, and wherein the new column elements of the updated first tile are different from column elements of the first tile; and the first tile and an updated second tile, wherein the updated second tile includes new row elements extracted from the second matrix, and wherein the new row elements of the updated second tile are different from row elements of the second tile. the instructions, when executed, further cause the computing device to store the first tile and the second tile in the dedicated memory. the instructions, when executed, further cause the computing device to receive, from the matrix multiplier unit, a subsequent partial dot product between: an updated first tile and the second tile, wherein the updated first tile includes new column elements extracted from the first matrix, and wherein the new column elements of the updated first tile are different from column elements of the first tile; and the first tile and an updated second tile, wherein the updated second tile includes new row elements extracted from the second matrix, and wherein the new row elements of the updated second tile are different from row elements of the second tile.

Images