Large parameter set calculation accelerator using configurable connectivity grids

Abstract

Methods and systems relating to a computer architecture are disclosed herein. A disclosed system includes a set of multipliers coupled to an input via an input, a connectivity grid, a set of readable cells, and a read circuit. Each readable unit of the set of readable units is uniquely associated with a parameter of the vector, the parameter having a value. Each readable unit in the set of readable units is also coupled via the connectivity grid to a multiplier from the set of multipliers that outputs a product of the value of the parameter and the input. The read circuit reads a product of the input and the vector from the set of readable units.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Patent No. 63 / 543,728, filed October 11, 2023, which is incorporated herein by reference in its entirety for all purposes.

[0002] background Machine learning is gaining significant importance in our society. Businesses are leveraging its power to extract actionable insights from massive datasets, automate tasks, and predict future trends with unprecedented accuracy. As a result, companies utilizing machine learning capabilities gain a competitive advantage, while the economy as a whole benefits from increased efficiency, productivity, and the creation of new markets and opportunities. However, this surge in activity has led to a substantial increase in the computing resources required and consumed by machine learning applications.

[0003] The increasing computational costs in machine learning applications pose a significant challenge to the field. As machine learning models become increasingly complex and datasets grow exponentially, the demand for computing resources also increases. For example, training deep neural networks can require substantial processing power and memory, often pushing the limits of existing hardware. This increased cost not only impacts the affordability of training models but also raises environmental concerns due to increased energy consumption. Furthermore, it creates barriers to entry for small organizations and researchers with limited resources. Addressing this challenge involves developing more efficient algorithms, improving parallelization software techniques, hardware acceleration, and many other approaches.

[0004] Machine learning computations typically require a large number of matrix multiplications. In fact, matrix multiplication is often a fundamental building block of machine learning computations across a wide range of models and applications. For example, in the world of fully connected neural networks, matrices represent the weights and activation outputs of interconnected neurons in each layer. These weights represent the model data that the neural network has learned during training. During training and inference, these matrices are repeatedly multiplied together to transform the input data into meaningful predictions. The essence of deep learning lies in its ability to learn complex patterns by fine-tuning these matrices using backpropagation, a process that also heavily relies on matrix multiplication. As models and datasets grow, the scale of required matrix multiplications increases accordingly, necessitating specialized hardware and optimization algorithms to ensure efficient and timely execution of machine learning tasks. Summary of the Invention

[0005] This paper discloses methods and systems relating to computational architectures. More specifically, this paper discloses methods and systems relating to computational architectures for accelerating vector multiplication.

[0006] As previously mentioned, the size of the data structures involved in machine intelligence computing is growing at a remarkable rate. Modern neural network models can contain over a trillion parameters, each used for multiple computations during model execution. These computations are typically matrix multiplications, where each independent matrix can contain billions of parameters. As a result, billions (if not trillions) of discrete multiplication operations need to be computed to perform matrix multiplications with that matrix.

[0007] While a single matrix can contain billions of parameters, the number of available values ​​for each parameter is relatively limited in many applications. This is because the number of bits assigned to represent each parameter value is typically finite, in order to reduce the memory and computational resources required to store and execute the model. For example, many accelerator architectures for machine learning applications utilize 8-bit data types, which can be used to represent up to 256 distinct values. Some accelerator architectures even utilize 4-bit data types, which can be used to represent up to 16 distinct values. Therefore, given a matrix with billions of parameters, it is easier to multiply each possible value of a matrix parameter by a number than to multiply each parameter in the matrix by a number. From a traditional computational perspective, this is an unconventional case, as the potential values ​​of unknown parameters are usually far greater than the number of unknown parameters in computation. However, this different way of thinking about the requirements of large matrix computations forms the basis of one of the many improvements to the systems and methods disclosed in this paper.

[0008] In a specific embodiment of the invention, a system for vector multiplication is disclosed. The system may include a set of multipliers, a connectivity grid, and a set of readable units. The set of multipliers may be hardwired to multiply each possible value of a vector's parameter by an input value. Each multiplier in the set may be associated with a value from the set of each possible value of the vector's parameter in a one-to-one correspondence. That is, each multiplier may be configured to receive an input value for computation and a unique value from the set of each possible value of the vector's parameter, treating those values ​​as operands of the multiplication. The connectivity grid may link the outputs of the set of multipliers to the set of readable units. Each readable unit may be associated with a parameter of the vector and connected via the connectivity grid to the output of the multiplier associated with the value of that parameter. The value obtained by multiplying the vector by the input value can then be read from the set of readable units. This read operation causes the value of the output vector to be read in parallel, the value of which is equal to the vector multiplied by the input value used for computation.

[0009] In a particular embodiment, the input value may be part of another vector and may be provided serially to the system to aid in calculating the multiplication of the two vectors. This result can be achieved by accumulating the output value read from the readable unit after each read operation.

[0010] In a particular embodiment, the system for vector multiplication can be part of a system for matrix multiplication. The set of readable units can be divided into subgroups of readable units, each subgroup being uniquely associated with a column or row of a matrix. The subgroups of readable units can be addressed and read independently, such that the output matrix can be assembled from a series of output values ​​read from the subgroups of readable units.

[0011] In certain embodiments, the connectivity grid can be configurable. For example, the connectivity grid may include a set of programmable switches that connect the output of a set of multipliers to the set of readable cells. The switches can be configurable during system deployment. For example, the switches can be controlled by delivering control signals to the control nodes of the switches once the system has been built and powered on. Alternatively, the switches can be configurable while the system is being built. The state of the switches can be set using methods similar to storing values ​​in read-only memory (ROM). The state of the switches can be set by activating specific transistors through controlled dopant delivery, by using a custom layout of the device's wiring layers, or by establishing or disconnecting connections between different circuit nodes using fuses or antifuses.

[0012] The configurability of a connectivity grid allows a system to be used as a computational structure for a specific matrix. For example, in machine intelligence applications, model data can be used to configure the connectivity of the grid, enabling the system to be customized for that model. Since the model data remains unchanged after training, a trained system can be deployed to such systems to provide an efficient means of executing the model, while the same system can be configured for different trained models by simply changing the system's connectivity grid. For instance, an integrated circuit can be configured to accelerate inference for different trained models by reconfiguring its configurable grid for different trained models (e.g., in the same way that multiple integrated circuits with the same underlying design can be programmed to have different ROM values). In certain embodiments, the connectivity grid of a system can be configured using methods for mask ROMs, such as by using different high-layer metal wiring masks customized for a specific model. Modifying a system for a specific model may essentially require storing the model in the system's mask ROM.

[0013] In a particular embodiment, the multiplier outputs a digital or analog value representing the output of the multiplication operation. The analog value may be represented by different reference voltages extending from ground to the supply voltage. The digital value may be a serialized pulse sequence or any other type of digital encoding. The group of multipliers may include a serializer. The read circuitry for the group of readable units may include a deserializer. The group of multipliers may be a bit-serialized multiplier. The read circuitry for the group of readable units may be coupled to a bit-serialized adder or multiplier. Digital signals may be continuously transmitted to the readable units such that once those readable units are read, the digital signals can be read from the unit. Alternatively, digital signals may be transmitted to the unit simultaneously with reading the unit to reduce unnecessary signal transmission through the connectivity grid. For example, a read command for a readable unit may simultaneously trigger all multipliers to output a serialized digital value, which is transmitted through the connectivity grid to the readable unit and then to the reader.

[0014] In specific embodiments of the invention, readable cells can be read in various ways. For example, a set of readable cells associated with a vector can be read in parallel to obtain a result of multiplying all values ​​in the vector by the input values ​​in a single read operation. As another example, a set of readable cells associated with a vector can be read multiple times in parallel, applying different values ​​as input values, to obtain a result of multiplying all values ​​in the vector by those input values ​​in a series of read operations. In a specific embodiment, each set of outputs read from the set of readable cells can be applied to an accumulator that accumulates the outputs obtained during the sequence of read operations. As another example, the set of readable cells can include subgroups of cells associated with different vectors. The subgroups of cells can be organized such that they can be read independently. For example, the set of readable cells can be configured similarly to a memory array, and each subgroup of cells can be associated with a separate word line, allowing all values ​​associated with that subgroup to be read in parallel.

[0015] In a particular embodiment of the invention, a system is provided. The system includes: a set of multipliers coupled to an input via an input; a connectivity grid; a set of readable cells, wherein each readable cell in the set of readable cells: (i) is uniquely associated with a parameter of a vector having a value; and (ii) is coupled via the connectivity grid to a multiplier from the set of multipliers, the multiplier outputting the product of the parameter value and the input; and a readout circuitry that reads the product of the parameter value and the input from the set of readable cells.

[0016] In a particular embodiment of the invention, a method is provided. The method includes: providing a set of readable cells, wherein each readable cell in the set is uniquely associated with a parameter of a vector having a value; and configuring a connectivity grid such that each readable cell in the set is coupled via the connectivity grid to a multiplier in a set of multipliers, the multiplier outputting the product of the parameter value and the input.

[0017] In a particular embodiment of the invention, a method is provided. The method includes: applying an input to a set of multipliers, wherein the set of multipliers is coupled to a set of readable cells via a connectivity grid, and each readable cell in the set of readable cells: (i) is uniquely associated with a parameter of a vector having a value; and (ii) is coupled via the connectivity grid to a multiplier from the set of multipliers, the multiplier outputting the product of the parameter value and the input; and using readout circuitry to read the product of the parameter value and the input from the set of readable cells. Brief description of the attached diagram The accompanying drawings illustrate various embodiments of the systems, methods, and other aspects of the present invention. Those skilled in the art will understand that the element boundaries (e.g., frames, groups of frames, or other shapes) depicted in the drawings represent one example of a boundary. In some examples, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some examples, an element shown as an internal component of one element may be implemented as an external component in another example, and vice versa. Furthermore, elements may not be drawn to scale. A non-limiting and non-exhaustive description is provided with reference to the following drawings. The components in the drawings are not necessarily drawn to scale, but are intended to illustrate principles.

[0019] Figure 1 A block diagram of a readable cell array according to a specific embodiment of the invention disclosed herein is illustrated, wherein a group of readable cells can be read in parallel.

[0020] Figure 2 A block diagram illustrates a readable cell array coupled to a side table of values ​​via a connectivity grid, according to a specific embodiment of the invention disclosed herein.

[0021] Figure 3 A block diagram of a matrix multiplication system with "2" as an example input, according to a specific embodiment of the invention disclosed herein, is illustrated.

[0022] Figure 4 An example layout of a portion of a readable cell and a connectivity grid according to a specific embodiment of the invention disclosed herein is illustrated.

[0023] Figure 5The diagram illustrates a layout of a readable unit and a portion of a connectivity grid according to a specific embodiment of the invention disclosed herein.

[0024] Figure 6 A set of word lines and bit lines of a programmable unit according to a specific embodiment of the invention disclosed herein are illustrated.

[0025] Figure 7 The matrix multiplication based on the relevant techniques is illustrated.

[0026] Figure 8 The diagram illustrates the alternative matrix multiplication method based on the relevant techniques.

[0027] Figure 9 Matrix multiplication using a connected grid is illustrated.

[0028] Figure 10 The illustration depicts a process of accelerating matrix multiplication using a multiplier with a single input, according to a specific embodiment of the invention disclosed herein.

[0029] Figure 11 The illustration depicts a process for accelerating matrix multiplication using a multiplier with two inputs, according to a specific embodiment of the invention disclosed herein.

[0030] Figure 12 The process of multiplying a vector by a line-readable cell is illustrated according to a specific embodiment of the invention disclosed herein.

[0031] Figure 13 The illustration depicts the process of multiplying a vector by two rows of readable cells, each row associated with a different connectivity grid arrangement, according to a specific embodiment of the invention disclosed herein.

[0032] Figure 14 The process of multiplying a vector by a multi-line readable cell and summing the results to produce an answer vector is illustrated according to a specific embodiment of the invention disclosed herein.

[0033] Figure 15 Examples of multipliers designed to multiply an input by a fixed value are illustrated according to specific embodiments of the invention disclosed herein.

[0034] Figure 16 An example of a configurable connectivity grid that connects the output of a multiplier to a corresponding readable unit, according to a specific embodiment of the invention disclosed herein.

[0035] Figure 17 An example block diagram is shown of a system for performing accelerated matrix multiplication according to a specific embodiment of the invention disclosed herein.

[0036] Figure 18 Examples of methods for accelerating matrix multiplication according to specific embodiments of the invention disclosed herein are illustrated.

[0037] Figure 19 Additional examples of methods for accelerating matrix multiplication according to specific embodiments of the invention disclosed herein are illustrated.

[0038] Detailed description Reference will now be made in detail to various aspects and variations of the systems and methods described herein, including implementations and examples. While several exemplary variations of the systems and methods are described herein, other variations may include aspects of the systems and methods described herein combined in any suitable manner, having all or some of the described aspects.

[0039] This document discloses in detail methods and systems relating to computer architecture. The methods and systems disclosed in this section are non-limiting embodiments of the invention, provided for illustrative purposes only, and should not be used to limit the full scope of the invention. It should be understood that the disclosed embodiments may or may not overlap with each other. Therefore, a part of an embodiment or a particular embodiment thereof may or may not be within the scope of another embodiment or a particular embodiment thereof, and vice versa. Different embodiments from different aspects may be combined or practiced individually. Many different combinations and sub-combinations of representative embodiments shown within the broad framework of the invention, which may be obvious to those skilled in the art but are not explicitly shown or described, should not be considered excluded.

[0040] In a particular embodiment of the invention, a system is provided. The system includes a set of multipliers input-coupled to an input, a connectivity grid, and a set of readable cells. Each readable cell in the set: (i) is uniquely associated with a parameter of a vector having a value; and (ii) is coupled via the connectivity grid to a multiplier from the set of multipliers, which outputs the product of the parameter value and the input. The system may also include readout circuitry that reads the product of the input and the vector from the set of readable cells. The system may be implemented in a circuit system. The system may be implemented as an integrated circuit or one or more integrated circuits communicating with each other. The system may be a matrix multiplication accelerator. The system may include a controller, such as a central processing unit, microcontroller, control circuitry, or other controller, capable of feeding operands to the matrix multiplication accelerator to accelerate computations involving large matrices.

[0041] In certain embodiments, the system according to the system described in the preceding paragraph can be used to accelerate computations involving inputs and vectors. In a first set of embodiments, the input can be a scalar value. The set of multipliers can each receive a scalar as input and different values ​​of the vector's parameters as operand inputs. The multipliers are coupled to the inputs and the different values ​​because those values ​​are delivered to the multipliers so that the multipliers use those values ​​as operands to perform multiplication. The different values ​​of the vector's parameters received by the multipliers can be all potential values ​​of the vector's parameters. Alternatively, in a second set of embodiments, the set of multipliers can each receive a scalar as input and be configured to multiply the input by a fixed value to output the product of the scalar and the fixed value. The fixed value can be all potential values ​​of the vector's parameters. In the second set of embodiments, the input can also be a vector, and the controller can be configured to sequentially provide the multipliers with independent scalar values ​​of the vector.

[0042] In certain embodiments, the same system may be part of a larger system for accelerating computations involving matrices, as described below. In these embodiments, the set of readable units may be organized into columns and rows, wherein each unit is uniquely associated with a parameter of the matrix. In these embodiments, controller circuitry may provide multiple input values ​​to the set of multipliers, use read circuitry to read the resulting output values, and accumulate the output values ​​in a specific pattern to produce the result of matrix multiplication. The controller circuitry may be configured to read certain subgroups of readable units in a specific order to obtain the result of matrix multiplication.

[0043] In certain implementations, the number of readable units will equal the number of parameters in the neural network matrix (e.g., the number of readable units in an integrated circuit implementation could be on the order of billions, where each readable unit is uniquely associated with one of billions of parameters). This set of readable units is associated with parameters in a one-to-one correspondence, and the set of readable units is ordered to match the order of the parameters in the vector. It should be noted that the readable units are not directly associated with the values ​​of the parameters, but rather with the parameters themselves. In other words, each readable unit is associated with an entry in the vector and is ordered to match the order of those entries.

[0044] In different applications, the radix of this set of multipliers can be set to different values. In a particular embodiment, the set of multipliers can be associated with each value in the vector in a one-to-one correspondence. For example, if the parameter is represented by a 4-bit data type, there will be a set of 16 multipliers (one multiplier for each unique value of the parameter), and if the parameter is represented by a 16-bit data type, there will be a set of 65,536 multipliers. In a particular embodiment, the parameter in the vector is an n-bit value, and the radix of the set of multipliers is 2 to the power of n.

[0045] In different applications, multiplier inputs can be provided in various ways. Multiplier inputs can be a set of fixed inputs and an input value. The set of fixed inputs can be the values ​​of parameters in the vectors to be multiplied, and the input value can be the value of the vectors to be multiplied. The input value can be provided by a controller. The set of fixed inputs can be hard-coded into the system's circuitry in various ways (making the input value the only true input to the multiplier). Voltage or current between zero voltage and supply voltage can be divided into a large number of values ​​to represent multi-bit values. Therefore, the set of multipliers can output a set of analog reference voltages or currents to the set of readable cells via a connectivity grid. The multiplier can be a bit-serial multiplier, and the readout circuitry can be coupled to an adder or accumulator also configured for bit-serial operation.

[0046] In addition to the circuitry that performs operand multiplication in various applications, multipliers may also include auxiliary circuitry. In a particular embodiment, the group of multipliers may include a set of serializers to serialize the output values ​​for transmission over a connectivity grid to readable cells. In these embodiments, read circuitry configured to read readable cells may include a set of deserializers to parse the values ​​after they have been read from the readable cells. In a particular embodiment, the group of multipliers may include a set of amplifiers to amplify the output values ​​for transmission over the connectivity grid. Amplifiers, serializers, or amplifiers and serializers may be tailored based on the number of readable cells connected to a given multiplier to provide appropriate amplification and thus optimize power consumption. For example, multipliers connected to a small number of readable cells may have lower amplification and power consumption to minimize power consumption, while multipliers connected to many readable cells may have higher amplification and higher power consumption to meet speed requirements. Tailoring may be performed in a calibration step, where the load on each multiplier is determined. This tailoring can improve the system's efficiency and power consumption.

[0047] Connectivity meshes can exhibit various properties in different applications. A connectivity mesh can connect the output of a set of multipliers to a set of readable units. A connectivity mesh can connect the output of a multiplier (which is associated with a given parameter value by taking that value as input) to a set of readable units, each associated with a parameter having that given parameter value. For example, a multiplier might have a fixed input value of 5, and the connectivity mesh could connect the multiplier's output to each readable unit associated with a parameter having the value 5. Based on the number of parameters in the vector with various values ​​associated with the multiplier, the connectivity mesh can have various fan-outs from the multiplier. In certain embodiments, the connectivity mesh can be configurable. The connectivity mesh can be configurable when the system is operating, when the system is being constructed, or during backend processing.

[0048] Readable units can exhibit various characteristics in different applications. In a particular embodiment, the group of readable units is configured to be read such that the output value of vector multiplication can be read in parallel and parsed by a higher-level system. Each readable unit can be associated with a parameter value from a vector. The group of readable units can represent a vector or a matrix. Individual readable units can be configured in various ways in different applications. In a particular embodiment, readable units can be configured similarly to ROM units. In a particular embodiment, readable units can consist essentially of access transistors. In a particular embodiment of the invention, the group of readable units can be addressed by read circuitry. The read circuitry can read the output value of a given vector multiplication in parallel.

[0049] In certain embodiments of the invention, an accumulator or parallel adder may be coupled to a readout circuit. When the readout circuit reads the product of an input value and a vector associated with a readable cell, the accumulator may add that product to the stored value. In this way, matrix multiplication can be facilitated by performing a multiplicative accumulation operation, since the readable cells are read sequentially and the resulting product values ​​are accumulated in the accumulator. The parallel adder may be a large adder that receives output vectors from several subgroups of readable cells and adds these values ​​together in parallel. The system may be configured to complete the multiplication and accumulation steps of matrix multiplication in a single clock cycle.

[0050] The matrix associated with the readable cell may include model data of the machine intelligence model, such that the values ​​do not change, and the system is configured to compute inference using a specific model. Assuming that in a particular embodiment, a back-end-of-line mask (such as a mask for the top metal layer) is used to configure the connectivity mesh, the resulting system typically provides an accelerator for the machine intelligence model, which can be customized for a specific machine intelligence model at a relatively low cost. This is because the cost of fabricating masks for different higher metal layers can be 1 / 100 to 1 / 10 of the cost of masks for the underlying circuitry system.

[0051] In specific machine intelligence applications, the matrix associated with the readable units mentioned in the previous paragraph can be extremely large in one dimension and relatively small in a second dimension (e.g., the second dimension is set by the layer's input). Therefore, the system according to this disclosure can be advantageously applied to machine intelligence applications because the second dimension determines the number of read operations that must be performed, which affects the time spent on computation, while the first dimension determines the size of the set of readable units. Given that the set of readable units can essentially consist of a single basic access transistor, this large first dimension can be made extremely high while maintaining low cost. Thus, the resulting system can perform machine intelligence matrix computations at high speed and low cost.

[0052] In a particular embodiment of the invention, a method for configuring a system to accelerate matrix multiplication is provided. The method includes providing a set of readable units, each of which is uniquely associated with a parameter of a vector having a value. The method further includes configuring a connectivity grid such that each of the readable units is coupled via the connectivity grid to a multiplier in a set of multipliers, the multiplier outputting the product of the parameter value and the input. In a particular embodiment, each of the readable units may be coupled via a conductive grid to a multiplier in a set of multipliers, the multiplier being coupled via the input to the parameter value. The method for configuring the system to accelerate matrix multiplication can be further used for steps involving accelerating matrix multiplication using the system. These steps may include applying an input to the set of multipliers and reading the product of the input and the vector from the set of readable units using readout circuitry.

[0053] In a particular embodiment of the invention, a method for accelerating matrix multiplication using a system is provided. The method includes applying an input to a set of multipliers. The set of multipliers is coupled to a set of readable cells using a connectivity grid, and each readable cell in the set is: uniquely associated with a parameter of a vector having a value; and coupled via the connectivity grid to a multiplier from the set of multipliers, the multiplier outputting the product of the parameter value and the input. The method further includes reading the product of the input and the vector from the set of readable cells using readout circuitry. The method for accelerating matrix multiplication using the system may precede steps for configuring the system for accelerating matrix multiplication. These steps may include configuring the connectivity grid before applying the input such that each readable cell in the set of readable cells is coupled via the connectivity grid to a multiplier in the set of multipliers, the multiplier being coupled via the input to the parameter value.

[0054] Figure 1 A block diagram of an array 101 of readable cells 102 is illustrated, wherein a group of readable cells can be read in parallel. In the illustrated example, the third row 103 of array 101 is being read, and all associated values ​​are being read from the cells in parallel. According to a specific embodiment of the invention, a group of readable cells can be associated with parameters in a matrix. In the illustrated configuration, the readable cells 102 are associated with a group of parameters of a 5:4 matrix, wherein each readable cell 102 stores the value of the associated parameter. Figure 1 In the readable unit 102, each item is marked with the value of the associated parameter stored by the associated readable unit. Output 104 indicates that the third line 103 is read as "1 1 2 3".

[0055] In a specific instance, array 101 may include model data of a machine intelligence model. This data may be part of a matrix multiplication operation using inputs and vectors. In a specific instance, the system according to this disclosure can be used to accelerate computations involving inputs and vectors. In a first set of instances, the input may be a scalar value. A set of multipliers may each receive a scalar as input and receive different values ​​of vector parameters as operand inputs. The multipliers may be input-coupled to the inputs and different values, as these values ​​are passed to the multipliers so that the multipliers use these values ​​as operands to perform multiplication. The different values ​​of vector parameters received by the multipliers may be all potential values ​​of the vector parameters. Alternatively, the set of multipliers may each receive a scalar as input and be configured to multiply the input by a fixed value by the product of the output scalar multiplied by that fixed value. The fixed value may be all potential values ​​of the vector parameters. In a second set of instances, the input may also be a vector, and the controller may be configured to sequentially provide independent scalar values ​​of the vector to the multipliers.

[0056] Figure 2 The diagram illustrates how side table 205 with value 206 can be used to generate a result similar to... Figure 1 The array contains the same information content (e.g., output 204 is the same as output 104). Figure 2 A block diagram of an array 201 of readable cells 202 is illustrated, wherein a group of readable cells 202 can be read in parallel. The values ​​of the readable cells 202 contained in array 201 are the same as the values ​​of the readable cells 102 contained in array 101. In the illustrated example, the third row 203 of array 201 is being read, and all associated values ​​are being read from the cells in parallel. In this case, side table 205 includes a set of values ​​206, which are equal to each potential value of the parameter in the matrix (e.g., each possible value of the readable cell 202). This set of values ​​206 includes the values ​​0, 1, 2, and 3. Next, side table 205 is connected to the readable cells 202 using a connectivity grid 207, such that the values ​​206 are connected to the readable cells 202 associated with the matrix parameter having these values. Therefore, when reading the readable cell 202 corresponding to the same parameter, the same value 206 is read from array 201 via the connectivity grid 207 connecting the readable cell 202 to its corresponding value 206. Using this method, side table 205 can be replaced with a set of multipliers, where the multipliers are connected to an input equal to the plotted latent value of the parameter in the matrix, and another input that can be used as a system input, to produce multiplication of the entire matrix by routing the values ​​to the array via connectivity grid 207. In a particular example, the multipliers are connected to the input that can be used as a system input, while the plotted latent value of the parameter is hard-coded (e.g., not an input).

[0057] Figure 3A block diagram of a system 300 for matrix multiplication is illustrated, including a set of multipliers 306, a connectivity grid 310, and an array of readable cells 312. The connectivity grid 310 may be configurable. The array of readable cells 312 may contain a large number of values. In the illustrated example, a portion of the fourth row of the array of readable cells 312 is being accessed (e.g., read), and all associated values ​​are being read from the cells in parallel. According to a specific embodiment of the invention, the array of readable cells 312 may be associated with parameters in a matrix or vector. A set 302 of possible parameters can be used to multiply the array of readable cells 312 with another matrix, vector, or scalar. The set 302 of possible parameters may be a set, array, list, or other data organization, or may be hard-coded into the multipliers 306. Figure 3 In the example, at least a portion of the array of readable units 312 is multiplied by a factor of 2. This factor of 2 may be a portion of the input vector, where system 300 is calculating the result of multiplying the input vector by the parameter matrix (e.g., via the array of readable units 312).

[0058] In this example as well, system 300 (e.g., an accelerator architecture for a machine learning system) can utilize a 4-bit data type, which can be used to represent up to 16 distinct values. Therefore, the set of possible parameters 302 includes a set of values ​​equal to each potential value of the parameter in the matrix, i.e., values ​​from 0 to 15. The set of possible parameters 302 is then multiplied by two (e.g., from the input vector) via a set of multipliers 306 to produce a set of multiplier outputs 308. The set of multipliers 306 can be connected to an array of readable cells 312 using a connectivity grid 310, such that the values ​​of the set of multipliers 306 are connected to the readable cells in the array of readable cells 312 associated with the matrix parameters having those values. Thus, the multiplier outputs 308 are input to the corresponding cells in the array of readable cells 312. For example, each cell in the array of readable cells having the value "a" can be connected to the corresponding "a" in the set of multipliers 306. Each cell in the array of readable cells with the value "b" can be connected to a corresponding "b" in a set of multipliers 306, and so on. The multipliers in the set of multipliers 306 are connected to an input equal to the plotted potential value of the parameter in the matrix (e.g., the set 302 of possible parameters), and another input (e.g., from an input vector) that can be used to generate multiplication of the entire matrix (e.g., the input vector being a part of it) by routing the potential value of the parameter through the connectivity grid 310 to the array of readable cells 312. The connectivity grid 310 can exhibit various characteristics in different applications. The connectivity grid 310 may contain a grid of wires and vias in the metal or wiring layers of one or more integrated circuits. The connectivity grid 310 may contain conductive connections for optical or electronic signals. The connectivity grid 310 may have various fan-outs from the multipliers in the set of multipliers 306, based on the number of parameters in a vector having various values ​​associated with these multipliers 306.

[0059] When reading the value of the array of readable units 312, the corresponding value of a set of multipliers 306 is read. For example, the value "a" of the array of readable units 312 will be read as "0" (which corresponds to the value "a" in the set of multipliers 306). The value "b" of the array of readable units 312 will be read as "2" (which corresponds to the value "b" in the set of multipliers 306). The value "c" of the array of readable units 312 will be read as 4 (which corresponds to the value "c" in the multiplier 306). The value "d" of the array of readable units 312 will be read as "6" (which corresponds to the value "d" in the multiplier 306). Therefore, when accessing the value "ab cd" from the array of readable units 312, the read circuit 314 outputs "0 2 4 6". In another example, if accessing unit "idkn" from the array of readable units 312 is desired, the read circuit 314 will output "16 6 20 26".

[0060] System 300 reduces the number of discrete multiplication operations performed when multiplying matrices. As illustrated, the array of readable cells 312 contains 16 possible values. If a row of parameter matrices is to be multiplied by two, each cell in a row of the array of readable cells 312 needs to be multiplied by two. If each cell in the array of readable cells 312 is directly multiplied by 2, a large number of multiplication operations will be performed. For example, if a row of the array of readable cells 312 contains 48 cells, there will be 48 multiplication operations. Instead of multiplying by each cell in a row of the array of readable cells 312, a set of multipliers 306 multiplies each possible value of the array of readable cells 312 by two, and then the result of the multiplication can be referenced when reading the corresponding row of the array of readable cells 312. Since there are only 16 possible values ​​for the readable cells, only 16 multiplications are performed. Sixteen multiplication operations can be performed much faster than 48 multiplication operations, and this benefit increases with the size of the array of readable units 312.

[0061] In different applications, the cardinality of a set of multipliers 306 can be set to different values. In a specific instance, the set of multipliers can be associated with each value in the vector in a one-to-one correspondence. For example, in a vector with 1,000 parameters and 57 unique values, the set of multipliers could be a set of 57 multipliers, where each multiplier is uniquely associated with one of the 57 unique values. In a specific instance, the set of multipliers can be associated with each potential value of a parameter in the vector (e.g., similar to a set of possible parameters 302). For example, in a vector with one trillion parameters, where each parameter can be one of 50,000 unique values, the set of multipliers could be a set of 50,000 multipliers, where each multiplier is uniquely associated with one of the 50,000 unique values. In a specific instance, the set of multipliers includes multipliers for each potential value of a parameter, where each potential value of a parameter is set by the resolution of the data type used to represent that parameter. For example, if the parameter is represented by a 4-bit data type, there will be a set of 16 multipliers, and if the parameter is represented by a 16-bit data type, there will be a set of 65,536 multipliers. In a specific instance, the parameter in the vector is an n-bit value, and the base of the set of multipliers is 2 to the power of n.

[0062] The array of readable cells 312 can be associated with model data of the machine intelligence model, ensuring that the values ​​do not change and that the system is configured to use that specific model for inference computation. It is assumed that, in a particular instance, the connectivity mesh 310 can be configured using back-end process masks (such as masks for the top layer metal). The resulting system typically provides an accelerator for the machine intelligence model, which can be customized for a specific machine intelligence model at a relatively low cost. This is because the cost of masks for fabricating different higher metal layers can be 1 / 100 to 1 / 10 of the cost of masks used for lower circuitry systems.

[0063] In certain machine intelligence applications, the array of readable units 312 may have a very high C value and a B value set by the input of this layer, which is typically kept relatively low. Therefore, a system based on the multiplication process described in the preceding paragraphs of this disclosure can be advantageously applied to machine intelligence applications because the B value sets the number of read operations that must be performed, which affects the time spent on computation, while the C value sets the size of the set of readable units. Assuming that the set of readable units can essentially consist of a single basic access transistor, this value can be made extremely high to reduce cost. Thus, the resulting system can perform machine intelligence matrix calculations at high speed and low cost.

[0064] Figure 4 An example embodiment of a readable cell 401 and a connectivity grid 405 in system 400 is illustrated. The readable cell 401 is essentially composed of an access transistor 402, with its gate connected to word line 403 and its source connected to bit line 404. The drain of access transistor 402 is connected to connectivity grid 405. As illustrated, connectivity grid 405 is configured such that readable cell 401 is associated with the value 2. Therefore, when bit line 404 and word line 403 of access transistor 402 are activated, the value read from readable cell 401 will be the value 2 (via a multiplier) multiplied by the value of input 406 of system 400. Word line 403 can be activated by sending a power supply voltage signal to access transistor 402, and bit line 404 can be selected by activating the read circuitry connected to bit line 404.

[0065] In specific instances, system 400 may be part of a larger system used to accelerate computations involving matrices. In these instances, a set of readable units 401 may be organized into rows and columns, with each readable unit 401 uniquely associated with a parameter of the matrix. In these instances, controller circuitry may provide multiple values ​​of input 406 to a set of multipliers, using read circuitry to read the resulting output values ​​and accumulate them in a specific pattern to produce the result of matrix multiplication. The controller circuitry may be configured to read certain subgroups of readable units 401 in a specific order to obtain the result of matrix multiplication.

[0066] The readable unit 401 can exhibit various characteristics in different applications. In a specific instance, a set of readable units 401 is configured to be read such that the output value of vector multiplication can be read in parallel and parsed by a higher-level system. The higher-level system logically links the application of specific input values ​​to this set of multipliers with the output values ​​read from the readable unit 401. Each readable unit 401 can be associated with a parameter value from a vector. The readable units 401 can be aggregated into a subgroup of readable units, where the subgroup of readable units represents the entire vector. A subgroup of readable units can be part of a larger set of readable units representing the entire matrix. The readable unit 401 can be part of a set of readable units that defines all available readable units in a given integrated circuit or a circuit system block within a given integrated circuit. For example, a given circuit system block may include an array of readable units 401, consisting of 1 billion readable units × 1,000 readable units, to store a matrix with one trillion parameters. The readable unit 401 can be distributed in various layouts relative to the group of multipliers. In a particular specific instance, the group of multipliers can be placed toward the center of a larger set of readable units so that the connectivity grid 405 can expand from the center of the block in multiple directions to minimize routing overlap.

[0067] Each readable unit 401 can be configured in various ways in different applications. In a particular specific instance, the readable unit 401 can be configured similarly to a ROM unit. In a particular specific instance, the readable unit 401 can essentially consist of an access transistor 402. When the access transistor 402 is turned on, the value routed from the multiplier to the readable unit 401 via the connectivity grid 405 can be accessed by the access transistor 402 and read by the read circuit. When the access transistor 402 is not turned on, the same read circuit can be used to read different values. The control node of the access transistor 402 can be connected to the word line 403. The drain or source node of the access transistor 402 can be coupled to the connectivity grid 405 and receive the output signal from the multiplier. Alternate drain or source nodes of the access transistor 402 can be coupled to the bit line 404. The bit line 404 can be connected to the read circuit. The readable unit 401 can be read by activating the word line 403 and the bit line 404.

[0068] Figure 5Five programming units (e.g., ROM programming units) according to specific embodiments of the invention disclosed herein are illustrated. All five programming units are formed from the same transistor. The transistor includes a drain 506 located in a region of the substrate, which has been doped to serve as the active region of the transistor. The drain 506 (e.g., the drain region) serves as a bit terminal of the transistor. The transistor also includes a source 508 located in a region of the substrate, which has been doped to serve as the active region of the transistor. The source 508 (e.g., the source region) serves as a read terminal of the transistor and is connected to a read line 510 of the ROM memory via a via 503. The transistor also includes a gate (e.g., coupled to a word line 512), which may be a strip of doped polysilicon or other conductive material above the channel of the transistor.

[0069] Figure 5 The transistors in the diagram have been programmed by vias 505 that connect bit lines 514 to bit terminals. In the illustrated configuration, via 505 is a first-layer metal via that connects the lowest metal layer forming the bit lines 514 to the active region of the transistor. As illustrated, three of the four bit lines 514 have been connected to the active region by introducing vias 505 at the intersection of the bit lines and bit terminals, while the fourth bit line has not yet been connected to the active region by introducing a via at that intersection. For clarity, Figure 5 The via 505 is shown off-center from the active region. However, the via 505 can be positioned at the center of the active region directly below the read line 510, since the read line 510 can be a second metal layer that can be routed over the first metal layer and its via (e.g., including via 505). Typically, to accommodate this configuration, the read line 510 can be any metal layer higher than the first metal layer, as long as the metal layer below it does not interfere with the via 503 connected to the read terminal of the transistor.

[0070] like Figure 5 As shown, the transistor read cell from the ROM involves a start word line 512 and one of a set of bit lines 514. The voltage required for the start word line 512 will depend on the polarity and characteristics of the transistor. However, regardless of the required voltage, the start word line 512 will result in a conductive path forming through the channel of the transistor, causing the bit terminal to be coupled to the read terminal. The voltage required for the start bit line (corresponding to bit line 514 of the read cell) will similarly depend on the polarity and characteristics of the transistor, and also on the characteristics of the read circuitry.

[0071] The characteristics of the read circuit will affect the voltage required to activate one of the bit lines 514. For example, activating one of the bit lines 514 may involve forming a current sink by lowering the voltage on the bit line to pull charge out of the read line 510 via a connection to the bit line. In these specific instances, when the voltage on the read line 510 does not drop in response to the read cell, the read circuit will be able to determine that no connection has been formed between the bit line and the active region, and thus read a first value from the memory. However, when the voltage on the read line 510 drops in response to the read cell, the read circuit will be able to determine that a connection has been formed between the bit line and the active region, and thus read a second value from the memory. As another example, activating one of the bit lines 514 may involve forming a current source by raising the voltage on the bit line to push charge onto the read line 510 via a connection to the bit line. In these specific examples, when the voltage on read line 510 does not rise in response to the read cell, the read circuit is able to determine that no connection has been formed between the bit line and the active region, and thus reads a first value from memory. However, when the voltage on read line 510 rises in response to the read cell, the read circuit is able to determine that a connection has been formed between the bit line and the active region, and thus reads a second value from memory. In specific examples where the cell has multiple values, the final voltage or the time taken to change the voltage on read line 510 can indicate the conductivity of the connection with the bit line, thereby reading multiple values ​​from memory.

[0072] In a specific instance, a transistor (e.g., similar to...) Figure 5 transistors or Figure 4 The access transistors (402) can be arranged in an array to form a ROM memory. Individual cells of the memory can be addressed using specific combinations of start word lines and bit lines. In a particular instance, the transistors and conductive lines forming the memory cells disclosed herein can be the same type of transistors and conductive lines used in any logic circuit system that can be implemented on the same substrate. Therefore, no additional processing steps are required to support the specific instance of the ROM disclosed herein. Instead, the mask used to form the bit terminals can be adjusted to account for the increased size of the transistors forming the RAM, while keeping all other processing steps the same. In an alternative approach, additional processing steps or a set of steps can be used to account for larger bit terminal sizes, such as by modifying the dopant dosage used to form the bit terminals (compared to the corresponding terminals of transistors in a logic circuit system on the same substrate).

[0073] Figure 5The readable cells illustrated can exhibit various characteristics in different applications. In a specific instance, a set of readable cells is configured to be read such that the output value of vector multiplication can be read in parallel and parsed by a higher-level system. The higher-level system logically links the application of specific input values ​​to a set of multipliers with the output values ​​read from the readable cells. Each readable cell can be associated with a parameter value from a vector. Readable cells can be clustered in a subgroup of readable cells, where the subgroup of readable cells represents the entire vector. A subgroup of readable cells can be part of a larger set of readable cells representing the entire matrix. A readable cell can be part of a set of readable cells that defines all available readable cells in a given integrated circuit or a circuit system block within a given integrated circuit. For example, a given circuit system block may include an array of readable cells (e.g., cells will resemble...). Figure 5 The array of readable cells consists of 1 billion readable cells × 1,000 readable cells to store a matrix with one trillion parameters. The readable cells can be distributed in various layouts relative to the set of multipliers. In a particular instance, the set of multipliers can be placed towards the center of a larger set of readable cells so that the connectivity grid can expand from the center of the block in multiple directions to minimize routing overlap. Individual readable cells can be configured in various ways in different applications. In a particular instance, readable cells can be configured similarly to ROM cells.

[0074] In a specific instance, Figure 5 It can be part of an array of readable cells or part of a connectivity grid (e.g., see...). Figure 6 For example, a transistor (e.g., an access transistor) can act as a switch. This switch can connect the outputs of a set of multipliers to the set of readable units. The switch can be configurable when the system is deployed. For example, the switch can be controlled by delivering control signals to its control nodes when the system has been manufactured and powered on. Alternatively, these switches can be configurable while the system is being manufactured. The state of the switch can be set using methods similar to those used to store values ​​in read-only memory (ROM). The switch can be a transistor or other form of hardware. The state of the switch can be set by controlling the delivery of dopant to activate a specific transistor, by using a custom layout of the device's wiring layers, or by using fuses or antifuses to establish or disconnect connections between different circuit nodes.

[0075] The configurability of a connectivity grid allows a system to be used as a computational structure for a specific matrix. For example, in machine intelligence applications, model data can be used to configure the grid's connectivity, enabling the system to be customized for that model. Since the model data remains unchanged after training, trained systems can be deployed to such systems to provide an efficient means of executing the model, while the same system can be configured for different trained models by simply changing the system's connectivity grid. For instance, integrated circuits can be configured by reconfiguring their configurable grids for different trained models to accelerate inference for different trained models (e.g., in the same way, multiple integrated circuits with the same underlying design can be programmed with different ROM values). In specific instances, methods for mask ROMs can be used to configure the system's connectivity grid, such as by using different high-layer metal wiring masks customized for a specific model. Modifying a system for a specific model may essentially require storing the model in the system's mask ROM.

[0076] Figure 6 A set of word lines and bit lines of a programmable (e.g., readable) unit as part of a system 600, according to a specific embodiment of the invention disclosed herein, are illustrated. System 600 may include a segment of ROM having eight transistors, which are connected to... Figure 5 The transistors and cells are consistent. As illustrated, different transistors are programmed and have different connection patterns between the bit lines and the bit terminals of the transistors. The diagram also shows how transistor columns share the same conductive lines for the bit lines and the same conductive lines for their gates. Furthermore, the diagram shows that the row spacing of the array is limited only by the minimum allowable distance between adjacent source and drain regions of different transistors, which is a smaller distance in modern processors. The diagram also shows how different columns of the ROM are separated by virtual gates, and how the source and drain regions of adjacent columns are placed close to each other.

[0077] The set of readable cells illustrated in System 600 includes rectangles of active semiconductor material forming the channels, sources, and drains of access transistors. The set of readable cells also includes vertical lines with diffused breaks (virtual gates) between adjacent columns to isolate adjacent access transistor stripes, vertical lines serving as word lines of the readable cells (e.g., on the lowest metal layer), and horizontal lines serving as bit lines (e.g., on the second metal layer). As illustrated, bit line contacts are shared by multiple readable cells because the "x" element indicates contacts for vias (e.g., between the metal and the semiconductor material in which the access transistors are formed). Also as illustrated, word lines are used to modulate the channels of the access transistors between an on state and a non-on state.

[0078] The set of readable cells in system 600 can be connected to a set of multipliers via a connectivity grid. The connectivity grid can be a set of metal layers that can be used to route various values ​​to the access transistors. In a specific instance, connecting a particular value to the access transistor involves adding jogs and via stacking from the metal associated with the desired value to the drain via contact of the access transistor. The connectivity grid (e.g., a grid) can be configured such that the number of potential values ​​that can be connected to a given access transistor is equal to the number of metal layers above the lowest two metal layers multiplied by the number of metal strips that can cover the readable cells, while allowing space for jogs from any strips.

[0079] During operation, Figure 6 The ROM in the diagram can be operated by precharging the read lines using the illustrated precharge buffer. Depending on the characteristics of the ROM, the read lines can be precharged to high, low, or tri-state values. Subsequently, a combination of word lines and bit lines can be initiated. A set of four values ​​can then be read from a column of ROM associated with the initiated word line. The bit lines will move charge to or out of the read lines, or not move charge to or out of the read lines, depending on the state of the cell (i.e., whether a connection is formed between the bit line and the bit terminal of the initiated cell). The voltage on the read lines can be monitored by a sense amplifier, which provides an output value on the illustrated output line after the read cycle. Therefore, the number of values ​​that can be read simultaneously is limited only by the number of available sense amplifiers and the number of transistors in the columns connected to these sense amplifiers.

[0080] The connection between the bit line and the bit terminal may include a diode. The diode can be used to prevent charge from moving through the connection to an unactivated bit line. For example, see reference... Figure 6 If the bit line associated with bit line buffer 601 is activated by pulling the bit line high, and the transistor is an n-type transistor with its drain as the bit terminal, care must be taken to ensure that current does not leak through the connection of the bit line associated with bit line buffer 602. Thus, the connection may include a diode, allowing current to flow to or out of the bit terminal only when the associated bit line is activated. The threshold voltage of the diode can be set high enough to prevent leakage while still being well below the differential voltage formed across the diode when the bit line is activated. In the case of the n-type transistor with a high-value activation bit line mentioned above, the transistor can be a pn junction transistor, where the n-type material is located on the bit terminal side of the junction, and the p-type material is located on the bit line side of the junction.

[0081] In a specific embodiment of the invention, the set of readable cells of system 600 can be addressed by read circuitry. The read circuitry can then retrieve the output value associated with the multiplication of a given vector from the readable cells. The read circuitry can address a given subgroup of readable cells to read the output value of a given vector multiplication in parallel. The read circuitry can be configured to read the same set of readable cells later when different input values ​​are provided to the multiplier, or to read a different set of readable cells later when the same input value has been provided to the multiplier, to obtain the results of various scalar-vector multiplications. The read circuitry can be configured to independently read a first subgroup of the set of readable cells associated with a first word line, and independently read a second subgroup of the set of readable cells associated with a second word line. The read circuitry can be configured to operate in a manner similar to that used in standard ROM, RAM, or flash memory, where different sets of memory cells are typically connected to word lines and can be read in parallel or individually with the aid of bit line selection.

[0082] In a particular embodiment of the invention, system 600 may include an accumulator. This accumulator may be coupled to a readout circuit. When the readout circuit reads the product of an input value and a vector associated with a readable unit, the accumulator may add the product to a stored value. In this way, matrix multiplication can be facilitated by performing a multiplicative accumulation operation, since the readable units are read sequentially and the resulting product values ​​are accumulated in the accumulator. In a particular embodiment of the invention, system 600 may include multiple readout circuits and accumulators such that a particular input can be multiplied simultaneously by individual vectors.

[0083] The readable cells in system 600 can include model data of a machine intelligence model, such that the values ​​do not change, and system 600 is configured to use that particular model to compute inference. The readable cells in system 600 can also be coupled to a set of multipliers via a connectivity grid. Assuming that in a particular instance, a back-end process mask (such as a mask for the top metal layer) is used to configure the connectivity grid, the resulting system typically provides an accelerator for the machine intelligence model, which can be customized for a specific machine intelligence model at a relatively low cost. This is because the cost of masks used to fabricate different higher metal layers can be 1 / 100 to 1 / 10 of the cost of masks used for the underlying circuitry system.

[0084] In certain machine intelligence applications, the readable units of system 600 are associated with a matrix having dimensions B:C. In a particular embodiment, the matrix has a very large C value and a B value set by the layer's input, which is typically kept relatively low. Therefore, system 600 can be advantageously applied to machine intelligence applications when performing matrix multiplication as described herein, because the B value sets the number of read operations that must be performed, which affects the time spent on computation, while the C value sets the size of the set of readable units. Assuming that the set of readable units can consist essentially of a single basic access transistor, this value can be made very high to reduce cost. Thus, the resulting system can perform machine intelligence matrix computations at high speed and low cost.

[0085] Figures 7 to 9 This can be used to illustrate matrix multiplication operations in specific embodiments of the present invention. Figure 7 The standard matrix multiplication is shown, where the multiplication of a matrix of size A:B with matrix B:C yields matrix A:C.

[0086] Figure 8 It shows the relationship with Figure 7 The same matrix multiplication (multiplying two matrices of dimensions A:B and B:C to produce matrix A:C) is decomposed into vector multiplication using each scalar from the columns of the first matrix with the vector rows of the second matrix, and then the products of these vector multiplications are accumulated.

[0087] Figure 9 It shows the relationship with Figure 7 and Figure 8 The same matrix multiplication (multiplying two matrices of dimensions A:B and B:C to produce matrix A:C). Figure 9 Use from Figure 8 Vector multiplication and scalars. However, Figure 9 A connected grid is implemented to improve the efficiency of matrix multiplication. The connected grid links identical values ​​(with the same padding, as shown) within each matrix 1:B. For example, the second and fourth cells of B1 are both "1". Instead of multiplying A1(5) by 1 twice separately, the system can use the calculated value of multiplying both the second and fourth cells of B1 by A1 (5, since 5 × 1 = 5), making C... i The second and fourth cells are 5. That is, A1 multiplied by 1 is calculated only once but used twice. The connectivity grid also connects the same values ​​in matrices B2, B3, and B4. The first and second cells of B2 ("1") are connected. The first, second, and third cells of B3 ("1") are connected. In the example of B4, no values ​​are repeated, and therefore no cells are connected. Although in Figure 9 In the example, only 1 is repeated in the 1:B matrix, but any value (0, 2, 3) can be repeated.

[0088] In a particular embodiment, Figure 9 The system can be used to multiply any two matrices of dimensions A:B and B:C by: partitioning the first matrix into A matrices of size 1:B and configuring a connectivity grid such that a set of readable cells is each connected to a multiplier associated with a value in the B:C matrix. Subsequently, the values ​​in each of the 1:B A matrices can be serially fed as input to the set of multipliers, and the values ​​from that set of readable cells can be read, allowing each row of the B rows of the B:C matrix to be read independently. While reading the values ​​produced by those B operations, these values ​​can be accumulated to obtain a 1:C vector (e.g., C1:B). i C ii C iii and C iv This can then be stored in a register, while the accumulator is used for the next operation. The process described in the preceding sentence can be repeated for each part of matrix A (A1, A2, A3, and A4) and the corresponding parts of matrix B (B1, B2, B3, and B4, respectively). Each resulting 1:C vector (C... i C ii C iii and C iv They can be added together to obtain a matrix of size A:C, which is equivalent to matrix multiplication of A:B by B:C.

[0089] Figure 9 The B:C matrix can include model data from the machine intelligence model, ensuring that the values ​​remain unchanged and the system is configured to use that specific model for inference computation. Assuming that in a particular embodiment, a back-end process mask (such as a mask for the top metal layer) is used to configure the connectivity mesh, the resulting system typically provides an accelerator for the machine intelligence model, which can be customized for a specific machine intelligence model at a relatively low cost. This is because the cost of masks used to fabricate different higher metal layers is 1 / 100 to 1 / 10 of the cost of masks used for the underlying circuitry.

[0090] In specific machine intelligence applications Figure 9 The B:C matrix has a very large C value and a B value set by the layer's input, which typically remains relatively low. Therefore, based on the use of this disclosure... Figure 9 The system of multiplication processes described herein can be beneficially applied to machine intelligence applications because the value of B sets the number of read operations that must be performed, which affects the time spent on computation, while the value of C sets the size of the set of readable units. Assuming that the set of readable units can essentially consist of a single basic access transistor, this value can be made very high to reduce cost. Therefore, the resulting system can perform machine intelligence matrix calculations at high speed and low cost.

[0091] Figure 10 A system 1000 for accelerating computations involving matrices is illustrated. System 1000 includes a multiplier 1014 with a single input. System 1000 performs a multiplication of a readable unit 1006 with an input 1002 using possible parameter values ​​1004 and a connectivity grid 1012. System 1000 multiplies the possible parameter values ​​1004 with the input 1002 and maps the corresponding results (via the connectivity grid 1012) to the readable unit 1006 to produce a result 1016. In certain embodiments, a set of readable units (such as readable unit 1006) may be organized into rows and columns, where each unit is uniquely associated with a parameter of a matrix (such as parameter 1010 of vector 1008). In these embodiments, controller circuitry may provide multiple input values ​​(such as input 1002) to a set of multipliers (such as multiplier 1014), use readout circuitry to read the resulting output values, and accumulate the output values ​​in a specific pattern to produce the result of the matrix multiplication (such as result 1016). The controller circuitry can be configured to read certain subgroups of readable cells in a specific order to obtain the result of matrix multiplication. In a particular embodiment, input 1002 can be a scalar value. In a particular embodiment, input 1002 can be a vector, and the controller can be configured to provide independent scalar values ​​of the vector to the multiplier in sequence.

[0092] exist Figure 10In the example, each readable unit in the group of readable units 1006 is uniquely associated with a parameter 1010 of vector 1008. Vector 1008 consists of parameters {X, Y, W, Z}. In this example, the group of readable units 1006 of system 1000 (which is configured to use vector 1008 to accelerate multiplication) has a base of 4, and four distinct readable units A, B, C, and D are associated with parameters X, Y, W, and Z, respectively (as shown, matching fills in the units of A with X, B with Y, C with W, and D with Z). This is a simplified example intended to illustrate the principles of an embodiment of the invention. Systems similar to system 1000 may include any number of inputs, possible parameter values, readable units, multipliers, and values ​​in vectors. In a particular implementation, the number of readable units 1006 will be equal to the number of parameters in the neural network matrix (e.g., the number of readable units in an integrated circuit implementation may be on the order of billions, where each readable unit is uniquely associated with billions of parameters). Readable units 1006 and parameters 1010 are associated in a one-to-one correspondence, and the group of readable units 1006 is ordered to match the order of parameters 1010 in vector 1008. It should be noted that readable units 1006 are not directly associated with the values ​​of parameters 1010, but rather with the parameters 1010 themselves. In other words, each readable unit 1006 is associated with an entry in vector 1008 and is ordered to match the order of those entries.

[0093] In system 1000, each readable unit 1006 is coupled via a connectivity grid 1012 to a multiplier 1014 in a set of multipliers, which outputs the product of the parameter value and the input. Multiplier 1014 can be a dedicated circuit that always outputs the number multiplied by the parameter value. In other words, multipliers can be programmed or constructed to always multiply a single input (such as input 1002) by a specific number (such as 0, 1, 2, etc.). Different multipliers can multiply the input by a specific number different from other multipliers in the system. For example, one multiplier can always multiply the input by 1, another multiplier can always multiply the input by 2, and so on. Instead of a multiplier that multiplies the input by 0, the input can also be connected to some other constant representation of ground or 0. The set of multipliers can each receive input 1002 (e.g., a scalar) and be configured to multiply input 1002 by a fixed value (0, 1, 2, etc.) to output the product of the scalar and that fixed value. The fixed values ​​can be all the potential values ​​of the parameters of vector 1008. In a particular embodiment, input 1002 can be a vector, and the controller can be configured to provide independent scalar values ​​of the vector to multiplier 1014 in sequence.

[0094] Vector 1008 consists of parameters {X, Y, W, Z}, where the values ​​of these parameters are {1, 2, 3, 4}, respectively. In this example, there are four multipliers 1014, which are hardwired or hardcoded to multiply by the values ​​1, 2, 3, and 4, respectively. The same four multipliers 1014 are coupled via input to input 1002, which will be used to multiply the vector. In this example, readable unit A is associated with parameter X and coupled via a connectivity grid 1012 to multiplier 1014, which is constructed to multiply by 1. Continuing the example, readable unit B is associated with parameter Y and coupled via a connectivity grid to multiplier 1014, which is constructed to multiply by 2. The same pattern continues for readable units C and D. The four multipliers 1014 in this example form a set of multipliers 1014. In a particular embodiment, the set of multipliers 1014 will include a multiplier 1014 for each possible (e.g., potential) parameter value 1004. For example, if parameters X, Y, W, and Z have arbitrary numbers of digits, the set of multipliers will include 10 multipliers, where each multiplier is configured to multiply the input (e.g., input 1002) by one of the values ​​from zero to nine. In a practical implementation, the multiplier configured to multiply by zero will be connected to some other constant representation of ground or 0, because in this case the multiplication calculation is independent of the input and is always 0.

[0095] Figure 11 A system 1100 for accelerating computations involving matrices is illustrated. System 1100 includes a multiplier 1114 with two inputs. System 1100 performs a multiplication of a readable unit 1106 with input 1102 using possible parameter values ​​1004 and a connectivity grid 1112. System 1100 multiplies the possible parameter values ​​1104 with input 1102 and maps the corresponding results (via the connectivity grid 1112) to readable units 1106 to produce result 1116. System 1100 can be similar to system 1000, except that in system 1000 each multiplier 1014 has a single input (e.g., input 1002), while in system 1100 multiplier 1114 has two inputs (e.g., input 1102 and input possible parameter values ​​1104). In a particular embodiment, input 1102 may be a scalar value. In a particular embodiment, input 1102 may be a vector, and the controller may be configured to provide independent scalar values ​​of the vector to the multiplier in sequence.

[0096] Each readable unit 1106 is uniquely associated with a parameter 1110 of a vector 1108. Vector 1108 consists of parameters {X, Y, W, Z}. In this example, this set of readable units 1106 in system 1100 (which is configured to use vector 1108 to accelerate multiplication) has a base of 4, and four distinct readable units A, B, C, and D are associated with parameters X, Y, W, and Z respectively (as shown, matching fills in the units for A with X, B with Y, C with W, and D with Z). This is a simplified example intended to illustrate the principles of an embodiment of the invention. Systems similar to system 1100 may include any number of inputs, possible parameter values, readable units, multipliers, and values ​​in vectors. In a particular implementation, the number of readable units 1106 will be equal to the number of parameters in the neural network matrix (e.g., the number of readable units in an integrated circuit implementation may be on the order of billions, where each readable unit is uniquely associated with billions of parameters). Readable units 1106 are associated with parameters 1110 in a one-to-one correspondence, and this group of readable units 1106 is ordered to match the order of parameters 1110 in vector 1108. It should be noted that readable units 1106 may not be directly associated with the value of parameter 1110, but may instead be associated with parameter 1110 itself. In other words, each readable unit 1106 may be associated with an entry in vector 1108 and may be ordered to match the order of those entries.

[0097] In system 1100, each readable unit 1106 is coupled via a connectivity grid 1112 to a multiplier 1114 from a set of multipliers 1114, which outputs a product of a possible parameter value 1104 and an input 1102. Different multipliers can multiply the input 1102 by a specific number that is different from the other multipliers in the system. In other words, the possible parameter value 1104 can each be unique.

[0098] In a particular embodiment, each readable unit 1106 is coupled to a multiplier 1114 via a connectivity grid 1112, the multiplier being input-coupled to the value of the parameter associated with the readable unit. In other words, the multiplier 1114 may have two inputs: input 1102 and input from possible parameter values ​​1104. In a particular embodiment, the possible parameter values ​​1104 may be variable or configurable. In a particular embodiment, scaling can be implemented by providing values ​​to the multiplier 1114. The set of multipliers 1114 may each receive a scalar as input 1102 and different values ​​of the vector's parameters as operand inputs (e.g., possible parameter values ​​1104). The multiplier 1114 is input-coupled to one value (e.g., input 1102) and different values ​​(e.g., possible parameter values ​​1104) because those values ​​are delivered to the multiplier 1114 in sequence, and the multiplier 1114 uses those values ​​as operands to perform multiplication.

[0099] Vector 1108 consists of parameters {X, Y, W, Z}, where the values ​​of these parameters are {1, 2, 3, 4}. In this example, there are four multipliers 1114, which are input-coupled to the values ​​1, 2, 3, and 4, respectively. The same four multipliers 1114 are also input-coupled to input 1102, which will be used to multiply vector 1108. In this example, readable unit A is associated with parameter X and coupled via connectivity grid 1112 to multiplier 1114, which is input-coupled to 1. Continuing the example, readable unit B is associated with parameter Y and coupled via connectivity grid 1112 to multiplier 1114, which is input-coupled to 2. The same pattern continues for readable units C and D. The four multipliers 1114 in this example form a set of multipliers 1114. In a particular embodiment, the set of multipliers 1114 will include a multiplier 1014 for each possible (e.g., potential) parameter value 1104. For example, if parameters X, Y, W, and Z have arbitrary number of bits, the set of multipliers will include 10 multipliers, each input-coupled to one of the values ​​from zero to nine. In a practical implementation, multipliers input-coupled to zero can be connected to some other constant representation of ground or 0, because in this case the multiplication is independent of the input and is always 0.

[0100] Figure 12This illustrates how a specific embodiment of the system disclosed herein can multiply a vector 1216 with a readable unit 1212. The first row (row 1201) shows a set of readable units 1212 and a side table 1214 of values, where the values ​​include each potential value (e.g., 0, 1, 2, and 3) of a parameter associated with a readable unit 1212. The readable units 1212 have the same padding pattern as the elements of the side table 1214 to demonstrate that they can be connected via a connectivity grid 1210 (e.g., an element of the side table 1214 with a given value can be connected to the readable unit 1212 associated with that value).

[0101] The second line (line 1202) illustrates how the value of vector 1216 (5 in the illustrated example) is multiplied by a value in side table 1214 to obtain a scaled table 1215. The value of vector 1216 can be provided as input to a multiplier. In a particular embodiment, the multiplier can be constructed to multiply the value of vector 1216 by a specific value from side table 1214 (e.g., the specific value is not input to the multiplier). In a particular embodiment, the multiplier can take both the value of vector 1216 and a specific value from side table 1214 (e.g., the value 0, 1, 2, or 3) as input, where the value of side table 1214 can be programmable or changeable.

[0102] The third line (line 1203) shows how the connectivity grid 1210 can connect the values ​​of the scaling table 1215 to the corresponding values ​​in the first row of readable cells 1212. Figure 12 In this example, the first value "0" of side table 1214 corresponds to the 6th cell in the first row of readable cells 1212. Therefore, the first value "0" of scaling table 1215 (corresponding to the first value "0" of side table 1214) is mapped to the 6th cell in the first row of readable cells 1212. As another example, the third value "2" of side table 1214 corresponds to the 3rd, 7th, and 8th cells in the first row of readable cells 1212. Therefore, the third value "10" of scaling table 1215 (corresponding to the third value "2" of side table 1214) is mapped to the 3rd, 7th, and 8th cells in the first row of readable cells 1212. This pattern continues to be used for other values ​​of side table 1214, scaling table 1215, and readable cells 1212.

[0103] The connectivity grid 1210 can exhibit various characteristics in different applications. The connectivity grid can connect the outputs of a set of multipliers to a set of readable cells 1212. The connectivity grid 1210 may include a grid of wires and vias in the metal or wiring layers of one or more integrated circuits. The connectivity grid 1210 may include conductive connections for optical or electronic signals. The connectivity grid 1210 can connect the outputs of multipliers (which are associated with a given parameter value by taking it as input or by hard-coding it) to a set of readable cells 1212, each associated with a parameter having a given parameter value. For example, the multiplier may have a fixed input value of 3, and the connectivity grid 1210 can connect the output of the multiplier to each readable cell 1212 associated with a parameter having the value 3. Based on the number of parameters in the vector with various values ​​associated with the multipliers, the connectivity grid 1210 can have various fan-outs from these multipliers.

[0104] The last line (line 1204) illustrates how the word lines (e.g., rows) of this set of readable units 1212 can be read while providing the corresponding values ​​for vector 1216 from scaling table 1215. In certain embodiments, readable units 1212 rewrite the values ​​of scaling table 1215 into the readable units. In these embodiments, the original values ​​of readable units 1212 (from line 1201) may no longer be stored in readable units 1212. In other specific embodiments, readable units 1212 may retain their original values ​​(from line 1202) and may not be rewritten to store values ​​from scaling table 1215.

[0105] Using four (B) iterations of the process from row 1201 to row 1204, the full vector multiplication (e.g., vector 1216 with values ​​{5, 9, 3, 2}) can be computed by summing the values ​​read during each iteration. In this example, the number of multiplications required for a single row of readable cells 1212 is 4, while the total number of multiplications required to complete the entire vector multiplication is 16. Using a more typical matrix multiplication process, the number of multiplications required for a single row of readable cells would be 8, while the total number of multiplications required to complete the entire vector would be 32. As shown in this example, performing vector multiplication as described herein reduces the number of discrete multiplications required. Furthermore, dimension D may extend to millions, billions, or even larger, while the number of potential values ​​for the parameters in dimension D will remain stable (e.g., side table 1214 will remain small relative to dimension D). Therefore, the number of multiplications will be fixed at the number of potential values ​​for the parameters multiplied by the size of dimension B (e.g., 16), while the number of multiplications originally required would be equal to dimension B multiplied by dimension D (e.g., 4 billion).

[0106] In different applications, the cardinality of this set of multipliers can be set to different values. In a particular embodiment, the set of multipliers can be associated with each value in the vector in a one-to-one correspondence. For example, in a vector with 1,000 parameters and 57 unique values, the set of multipliers could be a set of 57 multipliers, where each multiplier is uniquely associated with one of the 57 unique values. In a particular embodiment, the set of multipliers can be associated with each potential value of a parameter in the vector. For example, in a vector with one trillion parameters, where each parameter can be one of 50,000 unique values, the set of multipliers could be a set of 50,000 multipliers, where each multiplier is uniquely associated with one of the 50,000 unique values. In a particular embodiment, the set of multipliers will include multipliers for each potential value of the parameter, determined by the resolution used to represent the data type of the parameter. For example, if the parameter is represented by a 4-bit data type, there will be a set of 16 multipliers, and if the parameter is represented by a 16-bit data type, there will be a set of 65,536 multipliers. In a particular embodiment, the parameter in the vector is an n-bit value, and the base of the set of multipliers is 2 to the power of n.

[0107] Figure 13 Examples of multiple portions of a connectivity grid that can be used when multiplying the corresponding portion of vector 1316 with two rows of readable cells 1312 are shown. The first row (row 1301) shows a set of readable cells 1312 and a side table 1314 of values, where these values ​​include each potential value (e.g., 0, 1, 2, and 3) of a parameter associated with the readable cells 1312. The readable cells 1312 have the same fill pattern as the elements of the side table 1314 to demonstrate that they can be connected via a connectivity grid 1310 (e.g., an element of the side table 1314 with a given value can be connected to the readable cell 1312 associated with that value). The readable cells 1312 can be similar to readable cell 1212, the vector 1316 can be similar to vector 1216, and the side table 1314 can be similar to side table 1214.

[0108] The value of vector 1316 can be multiplied by side table 1314, which is mapped to the corresponding value in readable cell 1312. The second row (row 1302) shows how the connectivity grid 1310 can connect the value of scaled side table 1331 to the corresponding value in readable cell 1312. In this example, the first value (5) of vector 1316 is multiplied by readable cell 1312 in the first row 1321. The connectivity grid 1310 can be similar to the connectivity grid 1210. The first value (0) of side table 1314 corresponds to the 6th cell in readable cell 1312 in the first row 1321. Therefore, the value (0) of scaled side table 1331 corresponding to the first value (0) of side table 1314 and the input (5) from vector 1316 is mapped to the 6th cell in readable cell 1312 in the first row 1321. The second value (1) of the side table corresponds to the 2nd and 4th cells in readable cell 1312 in the first row 1321. Therefore, the value (5) of the scaled side table 1331 corresponding to the second value (1) of side table 1314 is mapped to the second and fourth units in the readable unit 1312 of the first row 1321. This pattern continues to be used for the third and fourth values ​​of side table 1314 and the first row 1321.

[0109] The third row (row 1303) shows how the connectivity grid 1320 can connect the values ​​of the scaling side table 1332 to the corresponding values ​​in the readable cells 1312. In this example, the second value (9) of vector 1316 is multiplied by the readable cells 1312 in the second row 1322. The first value (0) of side table 1314 corresponds to the 7th cell in the readable cells 1312 in the second row 1322. Therefore, the value (0) of the scaling side table 1332 corresponding to the first value (0) of side table 1314 and the input (9) from vector 1316 is mapped to the 7th cell in the readable cells 1312 in the second row 1322. The second value (1) of side table 1314 corresponds to the 1st, 2nd, 5th, and 6th cells in the readable cells 1312 in the second row 1322. Therefore, the value (9) of the scaled side table 1332 corresponding to the second value (1) of side table 1214 is mapped to the first, second, fifth, and sixth units in the readable units 1312 of the second row 1322. This pattern continues to be used for the third and fourth values ​​of side table 1314 and the second row 1322. This pattern also continues to be used for the values ​​of side table 1314 and the other readable units 1312 of the other rows.

[0110] Multiplying multiple rows of readable cells 1312 using four iterations can be computed by summing the values ​​read during each iteration (e.g., multiplying a vector 1316 with values ​​{5, 9, 3, 2} with readable cells 1312). In this example, the number of multiplications required for one row of readable cells 1312 is 4, while the total number of multiplications required to complete the entire vector multiplication is 16. Using a more typical matrix multiplication process, the number of multiplications required for one row of readable cells would be 8, while the total number of multiplications required to complete the entire vector would be 32. As shown in this example, performing vector multiplication as described herein reduces the number of discrete multiplications required. Furthermore, dimension D may extend to millions, billions, or even larger, while the number of potential values ​​for the parameters in dimension D will remain stable (e.g., side table 1314 will remain small relative to dimension D). Therefore, the number of multiplications will be fixed at the number of potential values ​​for the parameters multiplied by the size of dimension B (e.g., 16), while the number of multiplications originally required would be equal to dimension B multiplied by dimension D (e.g., 4 billion).

[0111] Figure 14 This illustrates how a specific embodiment of the system disclosed herein can multiply vector 1416 with multi-line readable unit 1412 and accumulate the results to produce result 1430. Figure 14 This illustrates how specific embodiments of the system disclosed herein can be performed with minimal multiplication. Figures 7 to 9 The operations shown (e.g., with) Figures 7 to 9 (Operations on matrices of the same dimension). Each column spanning rows 1402 and 1403 shows a multiplication associated with a row of readable cells. The first row (row 1401) shows a set of readable cells 1412 (associated with the values ​​of matrix B:C) and a side table 1414 of values, where these values ​​include each potential value of the parameter in B:C (e.g., corresponding to readable cell 1412). The readable cells 1412 have the same fill pattern as the elements of the side table 1414 to demonstrate that they can be connected via a connectivity grid (e.g., an element of side table 1414 with a given value can be connected to the readable cell 1412 associated with that value).

[0112] The second line (line 1402) illustrates how the value of vector 1416 can be multiplied by a value in side table 1414 (e.g., to obtain a scaled side table 1415). The value of vector 1416 can be provided as input to a multiplier. In a particular embodiment, the multiplier can be constructed to multiply the value of vector 1416 by a specific number (e.g., a specific number not input to the multiplier). In a particular embodiment, the multiplier can take both the value of vector 1416 and a value from side table 1414 (e.g., the value 0, 1, 2, or 3) as input, where the value of side table 1414 can be programmable or changeable.

[0113] The third row (row 1403) shows how the word lines (e.g., rows) of the set of readable cells 1412 can be read, while the corresponding values ​​of vector 1416 are provided to side table 1414 to obtain a scaled side table 1415 of that row multiplied by the vector values ​​of the matrix. Using four iterations of such a process (the four columns of rows 1402 and 1403), the full vector multiplication can be computed by accumulating the values ​​read during each iteration.

[0114] The fourth line (line 1404) illustrates the accumulation or summation step of matrix multiplication. The calculated values ​​corresponding to a column of readable cells 1412 are summed to produce a result 1430. Result 1430 has dimensions A:C. In a particular embodiment of the invention, the disclosed system may include an accumulator. This accumulator may be coupled to a read circuit. When the read circuit reads the product of a value from vector 1416 and a value associated with readable cells 1412 (possible parameter values ​​from side table 1414), the accumulator may accumulate the product with the stored value. In this way, matrix multiplication can be facilitated by performing a multiplicative accumulation operation, since readable cells 1412 are read sequentially and the resulting product values ​​are accumulated in the accumulator. In a particular embodiment of the invention, the disclosed system may include multiple read circuits and accumulators such that a particular input can be multiplied simultaneously with a single vector.

[0115] In a particular embodiment, the multiplier may be a bit-serial multiplier, and the readout circuitry may be coupled to an adder or accumulator also configured for bit-serial operation. The multiplier may provide signals to the connectivity grid in a bit-serial manner and transmit signals through the connectivity grid while calculating the signals. The accumulator and adder may receive signals from the connectivity grid via the readable unit 1412 and add or accumulate these values ​​upon receiving the signals. For example, a parallel adder may perform a bit-serial operation on all values ​​read from the readable unit 1412 to compute an output vector (e.g., result 1430) upon receiving bits from the readable unit 1412.

[0116] In specific embodiments of the invention, the disclosed system may include a parallel adder. The parallel adder may be a large adder that receives output vectors from several subgroups of readable units and adds these values ​​together in parallel. For example, the parallel adder may receive five large vectors of length X read from five subgroups of readable units and add the corresponding parameters of each of the five vectors in a single clock cycle to produce an output vector of length X (e.g., result 1430), which is equal to the sum of all those vectors. In this way, the system can be configured to perform the multiplication and accumulation steps of matrix multiplication in a single clock cycle. In specific embodiments, the parameters of a given matrix may be associated with more than one readable unit, wherein different readable units are each associated with more than one set of multipliers, so that multiple input values ​​can be multiplied with the parameters and accumulated in parallel in a single clock cycle. In specific embodiments, readable units associated with more than one vector can be read and added in parallel, so that a single input value can be multiplied with different vectors and accumulated in parallel in a single clock cycle.

[0117] exist Figure 14 In the example, the total number of multiplications required is 16, which is the same as... Figure 8 The total number of multiplications required is the same. However, this example is simplified because dimension C can extend to millions, billions, or even larger, while the number of potential values ​​for the parameters in dimension C will remain constant (e.g., side table 1414 will remain small relative to dimension C). Therefore, the number of multiplications will be fixed at the number of potential values ​​for the parameters multiplied by the size of dimension B (e.g., 16), while the number of multiplications originally required will be equal to dimension B multiplied by dimension C (e.g., 4 billion).

[0118] Figure 15 A multiplier 1514 is shown, designed to multiply input 1502 by a fixed value 1504. In certain embodiments, the multipliers 1514 may have different physical shapes because they are designed to multiply input 1502 by different fixed values ​​1504. However, the different multipliers 1514 may be designed to have the same footprint in layout so that they can be arranged compactly in an array. In a certain embodiment, the set of multipliers 1514 may be placed at the center of a large readable cell group so that the connectivity grid can spread from the center of the block in multiple directions to minimize routing overlap. The fixed value 1504 may be associated with each possible parameter of the matrix used in the matrix multiplication. Input 1502 may be associated with the value of a vector (e.g., a scalar value) used in the matrix multiplication.

[0119] Each multiplier 1514 can produce an output 1506 corresponding to the product of input 1502 and a fixed value 1504 corresponding to that multiplier 1514. Each output 1506 of the multiplier 1514 can be coupled (via a connectivity grid) to a readable unit. In a particular embodiment, the multiplier 1514 may form part of the connectivity grid. In other particular embodiments, the multiplier 1514 may be separate from the connectivity grid but still output to the connectivity grid. The multiplier 1514 may be a dedicated circuit that always outputs an input 1502 multiplied by a corresponding fixed value 1504. In other words, the multiplier 1514 can be programmed or constructed to always multiply a single input (such as input 1502) by a specific number (such as 0, 1, 2, etc.). Different multipliers 1514 can multiply input 1502 by a fixed value 1504 that is different from other multipliers 1514 in the system. For example, one multiplier 1514 may always multiply input 1502 by 1, another multiplier 1514 may always multiply input 1502 by 2, and so on. Instead of a multiplier that multiplies input 1502 by 0, input 1502 may also be connected to ground or some other constant representing 0. Input 1502 may be a scalar value. The fixed value 1504 may correspond to all potential values ​​of the parameters of the matrix used in matrix multiplication.

[0120] In a particular embodiment, the set of multipliers 1514 includes a multiplier 1514 for each possible parameter value of the matrix in matrix multiplication. Figure 15 In the example, the parameters of the matrix can have any value from 0 to 15. The set of multipliers 1514 includes 16 multipliers, each constructed to multiply the input 1502 by one of the values ​​from 0 to 15. In a practical implementation, the multiplier 1514 constructed to multiply by zero can be connected to ground or some other constant representing 0, because in that case the multiplication calculation is independent of the input 1502 and is always 0.

[0121] This set of fixed inputs can be hardcoded into the system's circuitry in various ways. For example, in... Figure 15 In this context, the fixed value 1504 can be integrated into the logic structure of each multiplier 1514 so that each multiplier 1514 is configured to receive only a single real input value (input 1502). Each multiplier 1514 may have slightly different connectivity or logic structure so that it is optimized for multiplication via its corresponding fixed value 1504.

[0122] In certain embodiments, the fixed value 1504 is not constructed into multiplier 1514. Instead, a value corresponding to the parameters of the matrix used in matrix multiplication can be input to the multiplier. In these embodiments, the multiplier has two inputs: an input value corresponding to the matrix parameters (corresponding to the fixed value 1504) and an input value for the vector (e.g., corresponding to input 1502). As another example, the fixed input can be stored in a hard-coded register and provided to the multiplier when the standard input is provided to the multiplier circuitry, with the exception that the same value and only that same value can be provided to one input of the multiplier circuitry.

[0123] In different applications, multiplier inputs can be provided in various ways. Multiplier inputs can be a set of fixed inputs (e.g., fixed value 1504) and input values ​​(e.g., input 1502). The set of fixed inputs can be values ​​of parameters in the matrices to be multiplied, and the input values ​​can be values ​​of vectors to be multiplied. Input values ​​can be provided by a controller. The controller can be a higher-order controller that coordinates the multiplication of two vectors or two matrices, where scalar vector multiplication is a single part of the multiplication.

[0124] The multiplier 1514 can take various forms in different applications. In a particular embodiment, the group of multipliers 1514 may include standard digital multiplier circuitry selected to operate with inputs of a data type having vector values. For example, the multiplier 1514 may be a 4-bit floating-point multiplier with two operand inputs. The multiplier 1514 may be a clock or static logic circuit. In a particular embodiment, the group of multipliers 1514 may be a mixed-signal or analog circuit block whose output represents an analog current or voltage signal of the multiplication output 1506. The voltage or current between zero voltage and the supply voltage can be divided into a large number of values ​​to represent multi-bit values. Therefore, the group of multipliers 1514 can output (e.g., as output 1506) a set of analog reference voltages or currents to the group of readable cells via a connectivity grid.

[0125] In addition to the circuitry for operand multiplication in various applications, the multiplier 1514 may include auxiliary circuitry. In a particular embodiment, the set of multipliers 1514 may include a set of serializers to serialize the output value (e.g., output 1506) for transmission over a connectivity grid to a readable cell. The serializers may serialize a digital value into a set of pulses, or an analog signal into a set of pulses with different amplitudes, such that each pulse is a multi-bit analog signal. In these embodiments, the read circuitry configured to read the readable cell may include a set of deserializers to parse the value after it has been read from the readable cell. In a particular embodiment, the set of multipliers 1514 may include a set of amplifiers to amplify the output value (e.g., output 1506) for transmission over a connectivity grid. Amplifiers, serializers, or amplifiers and serializers may be tailored based on the number of readable cells connected to a given multiplier 1514 to provide appropriate amplification and optimize power consumption. For example, a multiplier 1514 connected to a small number of readable units can have a lower amplification and power consumption to minimize power consumption, while a multiplier connected to a large number of readable units can have a higher amplification and higher power consumption to meet speed requirements. Trimming can be performed during a calibration step, in which the load on each multiplier 1514 is determined.

[0126] Multiplier 1514 can be a bit-serial multiplier, and the read circuitry can be coupled to an adder or accumulator also configured for bit-serial operation. Multiplier 1514 can provide signals to the connectivity grid in a bit-serial manner and transmit signals through the connectivity grid while calculating the signals. Accumulators and adders can receive signals from the connectivity grid via readable cells and add or accumulate values ​​upon receiving a signal. For example, a parallel adder can perform a bit-serial operation on all values ​​read from the readable cells to calculate an output vector while receiving bits from the readable cells.

[0127] Figure 16 A connectivity grid 1610 is shown connecting the output of multiplier 1614 and the corresponding readable unit 1606. The connectivity grid 1610 can exhibit various characteristics in different applications and is configurable. Multiplier 1614 includes multipliers 1615, 1616, 1617, and 1618. Readable unit 1606 includes units 1621, 1622, 1623, and 1624. The connectivity grid 1610 may contain a grid of wires and vias in the metal or wiring layers of one or more integrated circuits. The connectivity grid 1610 may contain conductive connections for optical or electronic signals.

[0128] In a particular embodiment, each multiplier 1614 may be associated with a given parameter value (A, B, C, or D) by hard-wiring the parameter to that multiplier 1614 (e.g., not as an input). A connectivity grid 1610 may connect the outputs of each multiplier 1614 to a subgroup of readable cells 1606 all associated with the given parameter value (A, B, C, or D). For example, multiplier 1618 may have a hard-wiring value 5 (corresponding to D), and connectivity grid 1610 may connect the output of multiplier 1618 to each readable cell associated with parameter D (e.g., having a value of 5). Based on the number of parameters in the vector having various values ​​associated with multipliers 1614, connectivity grid 1610 may have various fan-outs from those multipliers 1614.

[0129] In a particular embodiment, multipliers 1614 may each be associated with a given parameter value (A, B, C, or D) by taking that parameter value as input. Connectivity grid 1610 may connect the outputs of multipliers 1614 to subgroups of readable cells all associated with that given parameter value (A, B, C, or D). For example, multiplier 1618 may have a fixed input value of 5 (corresponding to D), and connectivity grid 1610 may connect the output of multiplier 1618 to each readable cell associated with parameter D (e.g., having a value of 5). Based on the number of parameters in the vector having various values ​​associated with multipliers 1614, connectivity grid 1610 may have various fan-outs from those multipliers 1614.

[0130] In certain embodiments, the connectivity grid 1610 may be configurable. The connectivity grid 1610 may be configurable when the system is in operation. For example, the connectivity grid 1610 may have a set of configurable switches 1630 that modify which readable units 1606 are connected to each multiplier 1614. The connectivity grid 1610 may be configurable when the system is being constructed. In embodiments where the system is implemented as an integrated circuit, the connectivity grid 1610 may be configurable when the set of transistors that will ultimately form the connectivity grid 1610 is formed by implanting diffusion implants into some transistors to turn on a first subgroup of switches 1630 and not implanting such implants into some other transistors to turn off a second subgroup of switches 1630. As another example, the connectivity grid 1610 may be configurable when the wiring layer of the integrated circuit is formed by coupling the outputs of the multiplier 1614 and the readable units 1606 using different wiring patterns. The connectivity grid 1610 may be formed in a metal layer of the integrated circuit. As another example, the connectivity grid 1610 can be configurable during post-processing. For example, a fuse or antifuse can be processed or subjected to high-energy applications to modify its conductivity state, thereby forming the connectivity grid 1610, where the connectivity between the output of multiplier 1614 and the readable cell 1606 differs. The connectivity grid 1610 can be configurable after the system has been constructed or during system operation. For example, switch 1630 can be a memory cell, such as static random access memory, flash memory, or other memory cells, which can be programmed to set the conductivity state of certain portions of the connectivity grid 1610 to connect specific multiplier outputs to specific readable cells (e.g., multiplier 1616 with cells 1622 and 1624, all corresponding to parameter D). As another example, switch 1630 may be a standard field-effect transistor that receives a control signal at its gate to set the conductivity state of certain portions of connectivity grid 1610 to connect a particular multiplier output to a particular readable cell (e.g., multiplier 1616 with cells 1622 and 1624, all of which correspond to parameter D).

[0131] The readable unit 1606 can be associated with model data of the machine intelligence model, such that the values ​​stored in the readable unit 1606 do not change, and the system is configured to use that particular model to compute inference. It is assumed that, in a particular embodiment, a back-end process mask (such as a mask for the top layer metal) is used to configure the connectivity mesh 1610. The resulting system typically provides an accelerator for the machine intelligence model, which can be customized for a specific machine intelligence model at a relatively low cost. This is because the cost of fabricating masks for different higher metal layers can be 1 / 100 to 1 / 10 of the cost of masks for the underlying circuitry.

[0132] In specific machine intelligence applications, the readable unit 1606 can have a very large number of columns and a number of rows defined by the input of this layer, which is typically kept relatively low. Therefore, matrix multiplication using the connected grid 1610 can be advantageously applied to machine intelligence applications because the number of rows in the readable unit 1606 determines the number of read operations that must be performed, which affects the time spent on computation, while the number of columns in the readable unit 1606 determines the size of the group of readable units 1606. Assuming that the group of readable units can essentially consist of a single basic access transistor, this value can be made extremely high to reduce cost. Thus, the resulting system can perform machine intelligence matrix calculations at high speed and low cost.

[0133] Figure 17 An example block diagram of a system 1700 performing accelerated matrix multiplication according to a specific embodiment of the invention disclosed herein is shown. In the matrix multiplication of system 1700, parameter matrix 1740 is multiplied by readable unit 1706. In a specific embodiment, system 1700 may be implemented on integrated circuit 1735.

[0134] The parameter matrix 1740 consists of various parameters with values. The parameter matrix 1740 can be split into distinct rows, where each row becomes a vector 1708. The number of distinct parameters that can make up the parameter matrix 1740 is finite, and the number of corresponding values ​​is also finite. The possible parameter value set 1704 contains each of these values. The parameter matrix 1740 can be n-bit (e.g., parameters can be n-bit values), and the possible parameter value set can have 2n... n The cardinality of the bits. The parameter matrix 1740 can be a neural network matrix.

[0135] Each vector 1708 can be further split into individual values. These vector values ​​can each be multiplied by a set of possible parameter values ​​1704 via a multiplier 1714. In a particular embodiment, the multiplier 1714 may include a serializer 1715, and the multiplier 1714 may be a bit-serial multiplier. The multiplier 1714 may be associated with each value in the vector 1708 in a one-to-one correspondence. The multiplier 1714 may be associated with each potential value of the parameter. The multiplier 1714 may output the product of the vector value and the possible parameter value as a product output 1712. The product output 1712 may be an analog reference voltage.

[0136] The connectivity grid 1710 can connect the product output 1712 to the corresponding readable unit 1706. The connectivity grid 1710 can be configured when the system 1700 is operating or when the system 1700 is being constructed. In a particular embodiment, the connectivity grid 1710 can be formed in a metal layer of the integrated circuit 1735.

[0137] In a particular embodiment, the number of readable cells 1706 will be equal to the number of parameters in parameter matrix 1740. For example, the number of readable cells 1706 in an integrated circuit embodiment may be on the order of billions, where each readable cell 1706 is uniquely associated with billions of parameters. This set of readable cells 1706 may be uniquely associated with parameters of vector 1708 or parameters of parameter matrix 1740. This set of readable cells 1706 and parameters are associated in a one-to-one correspondence, and the set of readable cells 1706 is ordered to match the order of parameters in parameter matrix 1740. It should be noted that the readable cells 1706 are not directly associated with the values ​​of the parameters, but rather with the parameters themselves. In other words, each readable cell 1706 is associated with an item in parameter matrix 1740 and is ordered to match the order of those items. In a particular embodiment, each readable cell 1706 may consist essentially of an access transistor.

[0138] In a particular embodiment of the invention, the set of readable units 1706 may be addressed by read circuitry 1720. In a particular embodiment, read circuitry 1720 may include deserializer 1721. Read circuitry 1720 may be used to retrieve mapped outputs 1717 (values ​​associated with the multiplication of vector 1708) from readable units 1706. Read circuitry 1720 may address a given subgroup of readable units to read multiple mapped outputs 1717 in parallel. Read circuitry 1720 may be configured to read the same set of readable units 1706 later when different input vector values ​​(from vector 1708) are provided to multiplier 1714, or to read a different set of readable units later when the same input vector values ​​(from vector 1708) have been provided to multiplier 1714, to obtain results of various scalar-vector multiplications. The read circuit 1720 can be configured to independently read a first subgroup of the group of readable cells 1706 associated with a first word line, and independently read a second subgroup of the group of readable cells 1706 associated with a second word line. The read circuit 1720 is configured to operate in a manner similar to that used in standard ROM, RAM, or flash memory, where different groups of memory cells are typically connected to word lines and can be read in parallel or individually with the aid of bit line selection.

[0139] In a particular embodiment of the invention, system 1700 may include an accumulator 1723. Accumulator 1723 may be coupled to read circuitry 1720. When read circuitry 1720 reads the mapped output 1717, accumulator 1723 may accumulate the value of mapped output 1717 with a stored value. In this way, matrix multiplication can be facilitated by performing multiplicative accumulation operations, because readable units 1706 are read sequentially and the resulting mapped output 1717 (e.g., the product value) is accumulated in accumulator 1723. In a particular embodiment of the invention, system 1700 may include multiple read circuits and accumulators such that a particular input can be multiplied simultaneously with a single vector.

[0140] In a particular embodiment of the invention, system 1700 may include adder 1722. Adder 1722 may be a parallel adder or a bit-serial adder. Adder 1722 may be a large adder that receives mapped outputs 1717 (e.g., output vectors) from several subgroups of readable units 1706 and adds these values ​​together in parallel. For example, adder 1722 may receive five large vectors of length X read from five subgroups of readable units 1706 (e.g., via read circuitry 1720) and add the corresponding parameters of each of the five mapped outputs 1717 in a single clock cycle to produce an output vector of length X (e.g., answer vector 1730) that is equal to the sum of all those mapped outputs 1717. In this way, system 1700 may be configured to perform the multiplication and accumulation steps of matrix multiplication in a single clock cycle. In a particular embodiment, a possible parameter value set 1704 (e.g., parameters of a given matrix) may be associated with more than one readable unit, wherein different readable units 1706 are each associated with more than one set of multipliers 1714, such that multiple input values ​​(e.g., corresponding to vector 1708) can be multiplied by the possible parameter value set 1704 and accumulated in parallel within a single clock cycle. In a particular embodiment, readable units 1706 associated with more than one vector 1708 can be read and added in parallel, such that a single input value can be multiplied by different vectors and accumulated in parallel within a single clock cycle.

[0141] In a particular embodiment of the invention, system 1700 may include control circuitry 1745 configured to provide a first address to the set of readable units 1706 when the possible input parameter value has a first value, to generate a first mapped output (e.g., a product) for read circuitry 1720, and to provide a second address to the set of readable units 1706 when the possible input parameter value has the first value, to generate a second mapped output (e.g., a product) for read circuitry 1720. Read circuitry 1720 may be coupled to accumulator 1723, which accumulates the first and second mapped outputs. Using this method, control circuitry 1745 can read a value for multiplying an input value by multiple vectors and accumulating them. Control circuitry 1745 may be configured to perform the above operations in the same manner as used by a microcontroller, core controller, or other control circuitry system when accessing a specific address in a memory array, wherein, in standard operation of a computation pipeline, input operands are delivered to the computation component.

[0142] In a particular embodiment of the invention, system 1700 may include control circuitry 1745 configured to provide a first address to the set of readable units 1706 when the possible input value 1704 has a first value, to generate a first product output for read circuitry 1720, and to provide a first address to the set of readable units 1706 when the possible input parameter value 1704 has a second value, to generate a second product output for read circuitry 1720. Read circuitry 1720 may be coupled to accumulator 1723, which accumulates the first and second mapped outputs. Using this method, control circuitry 1745 can read a value for multiplying two distinct input values ​​by a single vector and accumulating it. Control circuitry 1745 may be configured to perform the above operations in the same manner as used by a microcontroller, core controller, or other control circuitry system when accessing a specific address in a memory array, wherein, in standard operation of a computation pipeline, input operands are delivered to the computation component.

[0143] Figure 18 An example of a method 1800 for accelerating matrix multiplication according to a specific embodiment of the invention disclosed herein is shown. In a particular embodiment, steps or portions of steps of method 1800 may be rearranged, omitted, or repeated, or may otherwise deviate from the shown format.

[0144] At step 1802, a set of readable cells is provided. Each readable cell in this set can be uniquely associated with a parameter of the vector. This parameter can have a value.

[0145] At step 1804, a connectivity grid can be configured. The connectivity grid can be configured such that each readable unit in the set of readable units is coupled via the connectivity grid to a multiplier in a set of multipliers that outputs the product of the parameter value and the input. In a particular embodiment, the set of multipliers may be associated with each potential value of the parameter.

[0146] In a particular embodiment, at step 1806, the input can be applied to the group of multipliers.

[0147] In a particular embodiment, at step 1808, the product of the input and the vector can be read from the set of readable units. A read circuit can be used to read the product of the input and the vector. Method 1800 improves the efficiency of matrix multiplication, saves time and power, and has a lower manufacturing cost.

[0148] Figure 19 An example of a method 1900 for accelerating matrix multiplication according to a specific embodiment of the invention disclosed herein is shown. In a particular embodiment, steps or portions of steps of method 1900 may be rearranged, omitted, or repeated, or may otherwise deviate from the shown format.

[0149] In a particular embodiment, at step 1902, a connectivity grid can be configured. The connectivity grid can be configured prior to the application input (e.g., at step 1904). The connectivity grid can be configured such that each readable unit in a set of readable units is coupled via the connectivity grid to a multiplier whose output parameter value is the product of the input. This multiplier can be located within a set of multipliers.

[0150] At step 1904, the input can be applied to the set of multipliers. The set of multipliers can be coupled to the set of readable cells using a connectivity grid. Each readable cell in the set can be uniquely associated with a parameter of the vector. This parameter can be associated with a value (e.g., having this value). Each readable cell in the set can be coupled to a multiplier via a connectivity grid, which outputs the product of the parameter value and the input. This multiplier can originate from the set of multipliers.

[0151] At step 1906, the product of the input and the vector can be read from the set of readable cells. A read circuit can be used to read the product of the input and the vector from the set of readable cells. Method 1900 improves the efficiency of matrix multiplication, saves time and power, and has a lower manufacturing cost.

[0152] While this specification has been described in detail with respect to specific embodiments of the invention, it should be understood that modifications, variations, and equivalents of these embodiments will readily occur to those skilled in the art upon receiving the foregoing understanding. Any method steps discussed above can be executed by a processor operating in conjunction with a computer-readable, non-transitory medium storing instructions for those method steps. The computer-readable medium may be memory within a personal user device or network-accessible memory. Although the examples in this disclosure are generally directed to machine intelligence applications, the same methods can be used for other computationally intensive applications, including cryptographic computation, ray-tracing computation, etc. As another example, although the examples in this disclosure are generally directed to computations requiring multiplication of data structures with a number of parameters that are much larger than the potential values ​​of those parameters, the same methods can be used to perform different operations instead of multiplication, such as division, subtraction, addition, roots, logarithms, exponentiation, factorials, and any other mathematical or logical operations. These and other modifications and variations of the invention can be practiced by those skilled in the art without departing from the scope of the invention, which is more specifically set forth in the appended claims.

Claims

1. A system comprising: A set of multipliers, whose inputs are coupled to a single input; Connectivity grid; A set of readable units, wherein each readable unit in the set of readable units: (i) is uniquely associated with a parameter of a vector having a value; and (ii) is coupled via the connectivity grid to a multiplier from the set of multipliers, which outputs the product of the value of the parameter and the input; as well as A reading circuit that reads the product of the parameter value and the input from the set of readable units.

2. The system according to claim 1, wherein: The set of multipliers is associated with each value in the vector in a one-to-one correspondence.

3. The system according to claim 1, wherein: The set of multipliers is associated with each potential value of the parameter.

4. The system according to claim 3, wherein: The parameter is an n-bit value; and The base of the set of multipliers is 2 to the power of n.

5. The system according to claim 1, wherein: The set of multipliers includes a set of serializers; and The reading circuit includes a set of deserializers.

6. The system according to claim 1, wherein: The set of multipliers is a set of bit-serial multipliers; and The system also includes a set of bit serial adders coupled to the read circuit.

7. The system according to claim 1, wherein: The set of multipliers outputs a set of analog reference voltages to the set of readable units.

8. The system according to claim 1, wherein: The connectivity mesh is configurable when the system is in operation.

9. The system according to claim 1, wherein: The connectivity mesh is configurable when the system is being constructed.

10. The system according to claim 1, wherein: The system is implemented on an integrated circuit; and The connectivity grid is formed in the metal layer of the integrated circuit.

11. The system according to claim 1, wherein: The set of readable units can be addressed by the read circuit; and The read circuit is configured to: (i) independently read a first subgroup of the set of readable units associated with a first word line; and (ii) independently read a second subgroup of the set of readable units associated with a second word line.

12. The system according to claim 1, wherein: Each of the set of readable units is essentially composed of access transistors.

13. The system according to claim 1, further comprising: A parallel adder, coupled to the readout circuit, adds the product to a second product from the set of readable units.

14. The system according to claim 1, further comprising: An accumulator, coupled to the read circuit, accumulates the product of the input from the set of readable units and the vector with the stored value.

15. The system according to claim 1, further comprising: A control circuit is configured to: (i) when the input has a first value, provide a first address to the set of readable units to generate a first product for the read circuit; (ii) When the input has a second value, a second address is provided to the set of readable units to generate a second product for the read circuit; and An accumulator, coupled to the read circuit, accumulates the first product and the second product.

16. A method comprising: A set of readable units is provided, wherein each readable unit in the set is uniquely associated with a parameter of a vector, the parameter having a value; and Configure a connectivity grid such that each of the set of readable cells is coupled via the connectivity grid to a multiplier in a set of multipliers, which outputs the product of the parameter value and the input.

17. The method of claim 16, further comprising: Apply the input to the set of multipliers; and The reading circuit reads the product of the parameter value and the input from the set of readable units.

18. The method of claim 16, wherein: The set of multipliers is associated with each potential value of the parameter.

19. A method comprising: The input is applied to a set of multipliers, wherein the set of multipliers is coupled to a set of readable cells using a connectivity grid, and each readable cell in the set of readable cells: (i) is uniquely associated with a parameter of a vector having a value; and (ii) a multiplier coupled via the connectivity grid to a multiplier from the set of multipliers, which outputs the product of the value of the parameter and the input; and The reading circuit reads the product of the parameter value and the input from the set of readable units.

20. The method of claim 19, further comprising: Before applying the input, the connectivity grid is configured such that each of the set of readable cells is coupled via the connectivity grid to a multiplier in the set of multipliers that outputs the product of the parameter value and the input.

Images