Vector processing for in-memory computing

Abstract

The present invention relates, in particular, to an information processing device (10) which basically includes a vector processing unit (VPU) and an in-memory computing unit (IMC unit). The VPU (14) has vector registers designed to store vector components. The IMC unit (15) has a crossbar array structure adapted to store values ​​(called weights). The IMC unit is connected to the VPU's vector registers (148). The information processing device is configured to perform several operations, including vector transfer operations, vector supply operations, and vector read operations. During operation, the vector transfer operation causes the VPU to store the vector components of an input vector in its vector registers. Such components are generally fetched from adjacent memory (11) when the VPU receives the corresponding instruction from the main processing unit (12) to which it is connected. The vector supply operation supplies the input vector components from the vector registers as input to the IMC unit. This causes the IMC unit to perform matrix-vector multiplication based on the weights and input vector components to obtain the output vector accordingly. Finally, the vector read operation writes the output vector components of the output vector to the vector registers. From this point onward, the output vector components may be transferred to memory or the main processing unit (for further processing), or may be further processed in the VPU before being transferred to memory or the main processing unit. The present invention further covers the systems and methods relating to this invention.

Description

Technical Field

[0001] The present invention generally relates to the field of hardware-accelerated information processing, and more particularly, to the fields of in-memory computing and vector processing. More particularly, the present invention is directed to a method and system that relies on an in-memory computing unit having a crossbar array structure for efficiently performing matrix-vector multiplication, where the in-memory computing unit is connected to a vector processing unit to utilize the vector registers of the vector processing unit as a memory buffer.

Background Art

[0002] Artificial neural networks (ANNs), such as deep neural networks (DNNs), have revolutionized the field of machine learning by providing unprecedented performance in solving cognitive data analysis tasks. ANN computations mostly involve matrix-vector multiplication (MVM), which accounts for 70-90% of all neural network computations regardless of the ANN architecture. MVM computations pose multiple challenges due to their iterative nature, generality, computation, and memory requirements. Traditional computer architectures are based on the von Neumann computing concept, according to which processing power and data storage are divided into separate physical units. Such architectures have the drawbacks of congestion and high power consumption because data has to be continuously transferred from the memory unit to the control and arithmetic units through a physically constrained and costly interface.

[0003] One possible way to accelerate MVM is to use dedicated hardware acceleration devices, such as dedicated circuits with a crossbar array structure. This type of circuit includes input and output lines interconnected at intersections that define cells. Each cell contains a memory device (or set of memory devices) designed to store its respective matrix coefficients. Vectors are encoded as signals applied to the input lines of the crossbar array to perform MVM by multiplication-accumulation (MAC) operations. Such architectures can map MVM easily and efficiently. Weights are updated (in other words, replaced) by reprogramming memory elements to perform successive matrix-vector multiplications. Such techniques break the "memory wall" because the technique fuses arithmetic and memory units into a single in-memory computing (IMC) unit, thereby allowing processing to be performed much more efficiently in or near memory (in other words, the crossbar array). In addition, the IMC unit provides a solution to the von Neumann bottleneck on the instruction interface, as a single instruction may suffice to execute MVM over multiple cycles.

[0004] The primary computational load of ANNs such as DNNs is centered on MAC operations, but their execution involves additional operations for the IMC unit to communicate with external computerized entities, which slows down MVM execution. Therefore, the inventors addressed the challenge of further accelerating IMC calculations. [Overview of the project]

[0005] According to a first aspect, the present invention is embodied as an information processing device comprising a vector processing unit (VPU) and an in-memory computing unit (IMC unit). The VPU has vector registers designed to store vector components. The IMC unit has a crossbar array structure adapted to store values ​​(called weights). The IMC unit is connected to the VPU's vector registers. The information processing device is configured to perform several operations, including vector transfer operations, vector supply operations, and vector read operations. During operation of the device, a vector transfer operation causes the VPU to store the vector components of an input vector in its vector registers. Such components are typically fetched by the VPU from adjacent memory when it receives a corresponding instruction from the main processing unit. A vector supply operation causes the IMC unit to supply the input vector components from the vector registers (as input). This causes the IMC unit to perform matrix-vector multiplication (MVM) based on the weights and input vector components to obtain an output vector accordingly. Finally, a vector read operation writes the output vector components of the output vector to the vector registers. From this point onward, the output vector components may be transferred to memory or the main processing unit (for further processing), or they may be further processed in the VPU before being transferred to memory or the main processing unit.

[0006] According to the proposed solution, the VPU's vector registers are used as both source and destination buffers, thereby allowing vector components to be efficiently fed into and received from the IMC unit. That is, the pipelined parallelism enabled by the vector registers allows the IMC unit to operate more efficiently for executing MVM, not to mention the MVM acceleration inherently achieved through the IMC unit itself. Furthermore, as in the embodiment, the VPU's vector processing capabilities can be utilized to efficiently perform operations on vector components temporarily stored in the registers.

[0007] In some scenarios, directly loading the weight matrix from adjacent memory into the IMC unit offers little advantage compared to first loading the weights into a vector register. In such situations, the vector register can be utilized to store and modify the weights, which simplifies cell programming. Thus, in embodiments, the information processing device is further configured to perform weight transfer operations to store weights in a vector register and weight supply operations to supply weights from the vector register to the IMC unit, so as to store weights across a crossbar array structure. The device is preferably configured to perform the weight transfer and weight supply operations in stages, according to the memory capacity and / or throughput capability of the vector register.

[0008] Preferably, the crossbar array structure includes N × M cells, each having a memory system designed to store K weights (where K ≥ 2), thereby the crossbar array structure includes N × M memory systems adapted to store K sets of N × M weights, instead of a single set of N × M weights (as in the less preferred variant). In addition, the crossbar array structure includes a selection circuit configured for each memory system to enable N × M weights as active weights by selecting a weight from its K weights and setting the selected weight as the active weight. The objective is to reduce the idle time of the crossbar array structure (in other words, the core computing device), which is achieved by locally switching between active weights and thereby reducing the frequency of data exchange with the external memory unit. It should be noted that this selection may, in some cases, allow the entire matrix of N × M weights to be selected at once.

[0009] Furthermore, weights may be proactively loaded (in other words, prefetched during the computation cycle) to further reduce the idle time of the crossbar structure. That is, in a preferred embodiment, the information processing device is further configured to prefetch q sets of N×M weights while the IMC unit is performing MVM based on the N×M weights that are currently active, and to store the prefetched weights in N×M memory systems (where 1 ≤ q ≤ K-1) in place of q sets of N×M weights that are not currently enabled as active weights.

[0010] Preferably, the selection circuit includes N×M multiplexers, each connected to each of the K memory elements of an N×M memory system, and selection control lines connected to each of the multiplexers to allow any one of the K weights of each memory system to be selected and set as the active weight during operation.

[0011] In some embodiments, the vector processing unit further comprises an accumulation circuit interfaced with a vector register. The accumulation circuit may form part of, or be connected to, a component of the vector processing unit, such as a vector execution unit. The accumulation circuit is configured to accumulate the output vector components of an output vector obtained from an IMC unit. In this way, the output vector components of the output vector are accumulated in the vector register. This allows for the accumulation of the results of matrix-vector multiplications based on very large operands during operation, where such operands are mapped to inputs to the IMC unit and / or weights of the IMC unit. In variant forms, a similar accumulation circuit may be implemented in the IMC unit. However, it is preferable to leverage the processing capability of the vector processing unit by performing accumulation directly in the vector register rather than through an accumulation circuit provided in the IMC unit.

[0012] A vector register is designed to store a given number of numerical values. Preferably, the given number is greater than or equal to the smallest value of N and M. In this way, the weight values ​​stored in the IMC unit can be updated at least column by column or row by row. Note that in this context, updating weights means replacing at least some of the weight values ​​stored in the IMC unit with the new weight values ​​for the purpose of performing further MVM based on the new weight values. This type of update is independent of weight updates performed during the training of the underlying computational model, if such a model exists.

[0013] In a preferred embodiment, the information processing device further includes a controller, which preferably forms part of an IMC unit or is closely interfaced with an IMC unit. Furthermore, the VPU includes a vector instruction decoder connected to the controller, which is configured to decode vector instructions and transmit the decoded vector instructions to the controller. The controller is further configured to control vector supply operations, MVMs, and vector read operations based on the decoded vector instructions transmitted from the vector instruction decoder.

[0014] As mentioned above, the information processing device may be configured to store weights across a crossbar array structure. Vector instructions may form part of a vector instruction set designed according to a base instruction set architecture, where the base instruction set architecture can be advantageously extended with two types of vector instructions, each designed to store weights and perform MVM during the operation of the device.

[0015] In this embodiment, the information processing device further comprises a main processing unit configured to execute non-vector instructions. The main processing unit interfaces with a VPU to forward vector instructions to a vector instruction decoder.

[0016] In a preferred embodiment, the information processing device further comprises memory interfaced with a main processing unit and a VPU, respectively. The main processing unit further comprises an instruction decoder configured to fetch instructions from memory, decode the fetched instructions, and forward the vector instructions to the VPU accordingly. The VPU further comprises a vector load and storage unit configured to perform vector transfer and vector read operations (and, if present, weight transfer operations) by transferring corresponding values ​​between memory and vector registers during operation. The IMC unit is preferably integrated with the VPU. Alternatively, the IMC unit may be co-integrated with the VPU in the information processing device.

[0017] The present invention can also be embodied as an information processing system comprising one or more information processing devices as described above.

[0018] In another embodiment, the present invention is embodied as a method for operating an IMC unit to perform MVM. As described above, the IMC unit has a crossbar array structure for storing weights, and the IMC unit is connected to the vector register of the VPU. The method first includes performing a vector transfer operation to store the input vector components of an input vector in the vector register. Next, a vector supply operation is performed to supply the input vector components from the vector register as input to the IMC unit. Then, the method operates the IMC unit to perform MVM to obtain an output vector based on the stored weights and the input vector components supplied to the IMC unit. Finally, a vector read operation is performed to write the output vector components of the output vector to the vector register. Such steps are generally repeated to perform MVM sequentially.

[0019] Preferably, the method further includes performing a weight transfer operation to store the weights in a vector register and a weight supply operation to supply the weights to an IMC unit, so as to store the weights across the crossbar array structure, before performing the vector transfer operation and the vector supply operation. The weights may optionally be fetched directly from memory. In a variant, the weights are transferred through the vector register.

[0020] In a preferred embodiment, the crossbar array structure comprises N×M cells, each having a memory system designed to store K weights (K≧2), thereby the crossbar array structure comprises N×M cells, each containing an N×M memory system adapted to store K sets of N×M weights. In this case, the method may further include enabling the N×M weights as active weights by selecting a weight from its K weights for each memory system and setting the selected weight as the active weight, prior to performing the MVM. Weight transfer and weight supply operations are performed to store the K sets of N×M weights across the N×M memory systems of the crossbar array structure. Note that the weight transfer and weight supply operations may, in some cases, have to be performed stepwise, which depends on the memory capacity of the vector registers.

[0021] Such an architecture allows weights to be fetched proactively. That is, in a preferred embodiment, the method further includes prefetching q sets of N×M weights while the IMC unit performs MVM based on the currently active weights, and storing the prefetched weights in N×M memory systems instead of q sets of N×M weights that are not currently enabled as active weights (where 1 ≤ q ≤ K-1).

[0022] In an embodiment, the method further includes, in a VPU, decoding a vector instruction and executing the decoded instruction to control a weight transfer operation, a weight supply operation, a vector transfer operation, a vector supply operation, and a vector read operation. As described above, the vector instruction may form a part of a vector instruction set designed according to a base instruction set architecture extended with two types of vector instructions. The two types of vector instructions are each designed to cause a weight supply operation and an MVM to be performed. In a preferred embodiment, the method further includes sending a vector instruction from a main processing unit interfaced with the VPU before decoding the vector instruction in the VPU. The input vector components are preferably fetched from a memory interfaced with the VPU before performing the vector transfer operation so as to store the input vector components of this input vector in a vector register. Similarly, the output vector components can be transferred from the vector register to the memory after performing the vector read operation.

[0023] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of its exemplary embodiments, to be read in conjunction with the accompanying drawings. The drawings are for purposes of illustration only and will facilitate understanding of the invention by those skilled in the art together with the detailed description.

Brief Description of the Drawings

[0024] [Figure 1] FIG. schematically illustrates selected components of an information processing apparatus including a vector processing unit connected to an in-memory computing (IMC) unit according to an embodiment. [Figure 2A] FIG. schematically depicts a crossbar array structure of the IMC unit of FIG. 1 in an embodiment. [Figure 2B] FIG. schematically illustrates the integration of the IMC unit in the apparatus of FIG. 1. [Figure 3]A diagram of an IMC unit with a column of arithmetic units (multipliers and adder trees) connected to each column of memory elements. Each cell includes a memory system consisting of several memory elements in an embodiment. Overall, the crossbar array structure of the IMC unit includes N×M memory systems capable of storing K sets of N×M weights. [Figure 4] A diagram schematically depicting a given row of memory cells, as well as portions of a programming circuit and a selection circuit, in an embodiment. The depicted portions of the programming circuit and the selection circuit are connected to a single memory cell. Other portions of such circuits are not shown for the sake of simplicity of drawing. However, in reality, the programming circuit and the selection circuit are generally connected to each memory cell. [Figure 5] A simplified circuit diagram of components of a selection circuit connected to each memory cell in an embodiment. [Figure 6] A diagram schematically representing a computerized system with several devices according to an embodiment of the present invention. In this system, in an embodiment, it enables a user to interact with a server in order to accelerate computational tasks offloaded to the devices. [Figure 7] A flowchart illustrating high-level steps of a method of operating an IMC unit as depicted in FIGS. 2A - 3 so as to perform a matrix-vector multiplication according to an embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings show schematic views of the devices or portions thereof involved in an embodiment. Elements that are the same or functionally similar in the figures are assigned the same reference numerals unless otherwise specified.

[0026] Next, devices, systems, and methods embodying the present invention are described as non-limiting examples.

[0027] The following description is structured as follows: General embodiments and high-level variations are described in Section 1, and Section 2 deals with details of particularly preferred embodiments and technical implementations. Note that the Method and its variations are collectively referred to as the “Method.” All reference numerals Sn refer to method steps in the flowchart of Figure 7, and reference numerals relate to devices, components, and other concepts associated with embodiments of the present invention.

[0028] 1. General Embodiments and High-Level Modifications 1.1. Information Processing Device Next, a first aspect of the present invention will be described with reference to Figures 1 to 3. This aspect relates to an information processing device 10.

[0029] 1.1.1. Key Features As seen in Figure 1, the device 10 essentially includes a vector processing unit (VPU) 14 and an in-memory computing (IMC) unit 15. The VPU is a unit with vector processing capabilities. In particular, the VPU 14 includes vector registers 148, which together enable a vector register file. The vector registers 148 are designed to store vector components to be processed, in other words, arrays of numbers. In other words, the VPU is a type of computer processor that operates on vectors consisting of multiple elements instead of individual values. The IMC unit 15 includes a crossbar array structure adapted to store weights, as illustrated in Figures 2 and 3. Similar to the IMC unit, the crossbar array structure is indicated by reference numeral 15 in the drawings. The crossbar array structure 15 shown in Figure 2 includes N input lines 152 and M output lines 153, where N≧2 and M≧2. In practice, the number of input lines 152 and output lines 153 is generally in the range of several hundred to several thousand. For example, arrays of 256×256, 512×512, or 1024×1024 may be conceived, but N does not necessarily have to be equal to M. The input and output lines 152, 153 are interconnected at intersections (i.e., junctions), which define N×M cells 155. Each cell 155 contains its own memory system 157 (see Figure 4). Thus, the crossbar array structure 15 can store N×M weights. However, in a preferred embodiment, the IMC unit is designed to be able to store K distinct sets of N×M weights for reasons to be explained later (where K>1).

[0030] The IMC unit 15 is connected to the vector register 148 in order to efficiently use the vector register 148 as a memory buffer. That is, the information processing device 10 is configured to perform several operations, which enable the use of the vector register 148 to efficiently operate the IMC unit 15. Such operations essentially include vector transfer operations, vector supply operations, and vector read operations. During the operation, the vector transfer operation causes the input vector components of the input vector to be stored in the vector register 148. The vector supply operation causes the input vector components to be supplied from the vector register to the IMC unit 15 (as input). This enables the IMC unit to perform matrix-vector multiplication (MVM) based on the weights and input vector components. Such an operation causes the IMC unit to generate an output vector. Finally, a vector read operation is performed by the device 10 to write the output vector components of the output vector to the vector register 148.

[0031] By design, the IMC unit 15 performs MVM as a multiplication-cumulative (MAC) operation. Weights can be replaced by reprogramming memory elements when required to perform MVM. As stated in the background section, using the IMC unit 15 is advantageous because it breaks the "memory wall" and addresses the von Neumann bottleneck on the instruction interface, as discussed in the background section. Notably, further speedup is achieved here through the VPU 14. In fact, a vector element can be processed simultaneously across several processing elements, starting with the processing elements of the IMC unit 15. The vector element can be further processed by other components of the VPU, such as the vector execution unit, if necessary. Note that the vector element can be processed iteratively over several clock cycles, and the vector element can also be processed individually over consecutive clock cycles, as in the case of the scalar computing unit.

[0032] As with conventional VPUs, vector operations can be controlled by vector instructions, which may be incorporated into the normal instruction stream from the main processing unit (MPU) 12, as assumed in Figure 1. The required vector instructions may form an extension of the base instruction set architecture (ISA), as in the embodiments discussed in detail later. The MPU 12 is a set of neighboring processors that generally execute non-vector instructions. The MPU is generally a scalar or superscalar processing unit. The MPU may be a conventional processor, such as a central processing unit (CPU), or, as in the preferred embodiment, a customized processor with an integrated VPU 14.

[0033] With respect to the architecture, the VPU 14 may comprise several subunits and components, including the IMC unit 15 itself. That is, the IMC may be integrated as a component of the VPU 14, either interfaced with or integrated into the MPU 12 in a preferred embodiment. Thus, the device 10 shown in Figures 1 and 2B may, in some cases, be manufactured as an integrated structure, for example, a chip, that integrates all the components necessary to perform the computation. Such a component may additionally include memory 11 to which the MPU 12, VPU 14, and IMC unit 15 are connected, as assumed in Figures 1 and 2B. In variant forms, components 11-15 may be arranged as separate components of the same device or as several connected devices, but a subset of these components may be co-integrated. Thus, the technical term “device” used in reference to this information processing device 10 should be understood in a broad sense.

[0034] In this context, the IMC unit 15 generally consumes one data vector for each matrix-vector multiplication and generates one output vector. Thanks to the connection between the IMC unit 15 and the VPU register 148, the input vector can be transferred from memory 11 to the IMC unit 15 through the vector register 148. The output vector can also be written to the vector register 148 (in some cases, instead of the input vector pre-supplied to the IMC unit 15) before being transferred to other components 11, 12. In other words, the output vector components can finally be forwarded to adjacent memory 11 or MPU 12 for further processing, depending on the intended application. Such operation may require careful control, as will be described in detail below. The degree of control actually required depends on the memory capacity of register 148. A small register capacity will require frequent replacement of the same register. Alternatively, the output vector does not necessarily have to overwrite the input vector; instead, the output vector may be written to an additional vector register if the memory capacity of the vector register allows.

[0035] 1.1.2. Advantages of the proposed solution According to the proposed solution, the vector register 148 of the VPU 14 is used as both a source buffer and a destination buffer, thereby allowing vector components to be efficiently fed into and received from the IMC unit. That is, the pipelined parallel processing enabled by the vector register allows the IMC unit 15 to operate more efficiently for executing MVM, not to mention the MVM acceleration inherently achieved through the IMC unit 15 itself. Furthermore, as in the embodiment, the vector processing capabilities of the VPU 12 can be utilized to efficiently perform operations on vector components temporarily stored in the register.

[0036] 1.1.3. Preferred Embodiment 1.1.3.1. IMC Weight Storage and Update Firstly, the information processing device 10 may be configured to perform weight transfer and weight supply operations, which may take advantage of the vector register 148, as in the vector-related operations described above. Specifically, the weight transfer operation is performed to first store weights in the vector register 148, while the weight supply operation is performed to supply the weights stored in the vector register to the IMC unit 15. The ultimate goal is to store the weights across the crossbar array structure of the IMC unit 15 for subsequent MVM (Multiple Value Management) as needed. The memory capacity and / or throughput capability of the vector registers determine how such transfer operations are performed. Both factors (in other words, memory capacity and throughput capability) can be potential limiting factors, depending on the size of the crossbar array structure and how often weights need to be replaced. If necessary, weight-related operations are performed stepwise, in other words, in stages, within the limitations imposed by the above factors.

[0037] In some cases, weights can be stored in a single cycle, if the vector register capacity allows. Otherwise, weight calculations are performed incrementally. For example, weights can be supplied to the IMC unit sequentially column by column or row by row, and, if necessary, by portion, where the maximum portion size is determined by the number of vector registers 148. The same mechanism can be used to first store the weights and then update them, as needed to continuously perform MVM. In other words, the IMC unit 15 can continuously perform MVM based on the updated weights and the input vector components continuously supplied to the IMC unit 15.

[0038] In practice, the weight matrix does not require frequent updates, at least compared to the input vector. In some scenarios, directly loading the weight matrix from memory 11 into the matrix register of the IMC unit 15 offers little advantage over first loading individual rows or columns (or parts thereof) into the vector register 148. In such situations, the vector register 148 can be utilized to update the weights, which allows for simplification of cell programming. However, in other cases, directly loading the weights from memory 11 is more efficient. A unit of the VPU (e.g., unit 144 in Figure 1) can be connected to the programming means 158 of the IMC unit 15 (see Figure 4) to pass weight values. In a variant form, or as an addition, the controller 149 of the IMC unit 15 can be directly connected to memory 11, as indicated by the dashed arrow in Figure 1, to pass weight values ​​to the programming means 158. The device 10 is actually designed to provide both options, allowing for a choice between direct loading of weights and indirect transfer through the vector register. This allows the decision of whether to load weights directly from memory 11 to be automatically selected at runtime, depending on the characteristics of the computation to be performed. Furthermore, which option to use may be configured by the client, for example, when starting the job execution.

[0039] In general, the vector register 148 can be advantageously designed to store any number P of numerical values ​​(where P ≥ 2). Preferably, however, the register is sized such that this number P is greater than or equal to the number N, M of values ​​stored across a column or row of the IMC unit 15. In other words, P ≥ Min(N, M). In this way, the IMC unit 15 can update (at least) one column or row at a time.

[0040] 1.1.3.2. In-memory processing based on multiple weight sets Next, preferred embodiments are discussed with reference to Figures 2A to 4. Instead of storing a single set of N × M weights, the IMC unit 15 may be able to store several sets of N × M weights, which is advantageous in speeding up the weight matrix updates and facilitating very large MVM operations. This requires modifying the memory system 157 of the cell 155. That is, the crossbar array structure contains N × M cells 155, each having a memory system 157. Here, each memory system 157 may be designed to store K weight values ​​at once instead of a single value (where K ≥ 2). Each memory system 157 generally contains several memory elements, where each element is capable of storing one numerical value. In practice, K may be equal to, for example, 4, 8, 16, or 32 (as assumed in Figures 3 and 4). Overall, the crossbar array structure 15 includes N × M memory systems, each capable of storing K sets of N × M weights, in other words, a total of K × N × M weights.

[0041] It should be noted that the memory elements of each memory system can be broken down into several components, each of which stores a distinct bit value, and possibly a sign. However, each memory element should be capable of storing one true weight value at a time, so that each memory system can store K true weight values ​​at a time, and a crossbar can store K sets of N × M weight values. In other words, such a technique should be distinguished from, for example, techniques aimed at decomposing a single matrix into positive and negative parts. It should be noted that the above concept of a crossbar is, in practice, independent of the technique used in the memory system in each cell. In other words, this concept is not primarily dependent on a particular circuit implementation, and the weights can be stored using any suitable memory technique, such as static random access memory (SRAM), latches, or digital flip-flops, as well as resistive random access memory (RRAM) cells, although there are other examples. The purpose of storing K sets of N×M weight values ​​is to reduce the idle time of the core computing device, in other words, the crossbar array structure, which is achieved by locally switching between active weights and accordingly reducing the frequency of data exchange with the external memory unit.

[0042] Thus, in this embodiment, each cell is accompanied by a memory system 157 capable of storing K weights. Such weights are represented as W in Figure 2A. i,j,kIt is stated that i takes values ​​from 1 to N, j takes values ​​from 1 to M, and k takes values ​​from 1 to K. This allows some weights to be activated before performing MAC operations based on a given vector and matrix coefficients corresponding to the activated weights. A selection circuit 159 may be provided for each memory system 157 to select some weights from its K weights and set the selected weights as active weights. In this way, N × M weights can be activated as active weights. Note that the selection and setting of weights can actually be performed as a single operation when using a selection circuit that relies on a multiplexer 159 connected to each memory system, as in the embodiments discussed below with reference to Figures 4 and 5. Furthermore, such embodiments allow the entire matrix of N × M weights to be selected at once.

[0043] Once a set of weights is activated, the vector components can be supplied to the crossbar array structure 15 via a vector register. In this way, an input vector consisting of N components (also referred to herein as the "N vector") can be fed (as a signal) into the N input lines 152 of the crossbar 15, and the crossbar 15 performs MAC operations based on the N vector and the N × M active weights currently activated. The MAC operation results in each of the vector component values ​​supplied to the N input lines being multiplied by the currently active weight value. The crossbar configuration allows M MAC operations to be performed in parallel in each calculation cycle. Note that the operation performed in each cell actually corresponds to two scalar operations, in other words, one multiplication and one addition. Thus, M MAC operations imply N × M multiplications and N × M additions, totaling 2 × N × M scalar operations.

[0044] The output signals acquired on the M output lines 153 are then read out to obtain the corresponding values, which are buffered in the vector register 148. In practice, several computation cycles may be performed sequentially based on multiple N vectors and multiple weight matrices. The output values ​​of the MAC operation may actually correspond to partial values ​​(when very large operands are involved), and these partial values ​​may, in some cases, be accumulated in the IMC unit, in the outputs of M columns, or directly in register 148, as will be discussed in detail later.

[0045] As described above, the device 10 may be configured to perform weight transfer operations through the vector register 148. In a variant, the IMC may be programmed by directly fetching weights from adjacent memory 11. This variant is advantageous when a large set of weights needs to be stored in the IMC and continuously updated, and this can be done as a background task, in other words, behind the scenes, without interfering with input vector transfers. Furthermore, this allows for smooth prefetching of weights for subsequent computation cycles without affecting vector transfers, and thus without affecting core computations. That is, the device 10 may be further configured to prefetch q sets of N×M weights and store the prefetched weights in the N×M memory system 157 in place of q sets of N×M weights that are not currently enabled as active weights (where 1 ≤ q ≤ K-1). In other words, the q sets of N×M weights that are prefetched and stored in the IMC unit correspond to weights that are no longer active.

[0046] The prefetching operation can be performed proactively while the IMC unit 15 is performing the MVM based on the currently active N × M weights, which results in further speed improvements. That is, the device can proactively fetch and store weights that are planned to be used in subsequent MVMs. As mentioned above, the weights can be prefetched directly from memory in the background so as not to interfere with the current MVM operation as possible.

[0047] Notably, the above solution allows a separate set of weights to be activated locally in the crossbar array 15 in a timely manner, which reduces the frequency of data exchange with memory 11. This reduces the idle time of the IMC unit 15. In other words, up to K consecutive computation cycles can be performed without the need to transfer a new set of weights, thus avoiding idle time resulting from intermediate weight updates. Instead, the relevant weight set is activated locally as active weights when needed. Furthermore, the ability of weights to be proactively loaded (in other words, prefetched during computation cycles) further reduces the idle time of the crossbar structure 15.

[0048] 1.1.3.3. Accumulation As illustrated in Figures 3 and 4, columns of arithmetic units 156 can be placed next to any column of cells 155 of the IMC unit 15. Such arithmetic units can multiply the weights by the input vector values ​​and thus create partial products. All partial products are successively accumulated by the adder tree of units 156 to produce the result of the full dot product.

[0049] Additional accumulation may be performed to adapt to matrix-vector multiplication of very large operands, in other words, operands that cannot be directly adapted by the crossbar array. The K matrices of N×M elements provided by the crossbar array may, in fact, be insufficient to adapt to desired (which can be very large) operands, as is the case with complex machine learning tasks. For example, matrix-matrix multiplication with large operands (matrices) can also be handled by a smaller IMC array by locally activating each matrix coefficient array by decomposing the input matrix into input vectors, which are then divided into subvectors, and each subvector is assigned a separate block matrix to perform successive matrix-vector multiplications, thereby accumulating partial results directly, either in the vector register or in the IMC unit. Each final vector is then transferred to adjacent memory 11.

[0050] The required accumulation is preferably performed directly in the vector register 148, thanks to an accumulation circuit interfaced with the vector register 148. The accumulation circuit preferably forms part of unit 146, as discussed below. This accumulation circuit is configured to accumulate the vector component values ​​generated by the sequence of IMC units 15. Thanks to this accumulation circuit, the output vector components can be accumulated directly in the vector register 148. Therefore, in this case, there is no need to provide an accumulation circuit in the IMC unit, which provides further flexibility in the design of the IMC unit.

[0051] It should be noted that, in principle, such an accumulation circuit may be located on the path connecting the vector register 148 to the IMC unit's read unit. However, preferably, the accumulation circuit forms part of the VPU 14's vector execution unit 146. That is, such accumulation operations can be advantageously performed at the level of the vector register 148 or in parallel near the vector register 148 by leveraging the computing power of the vector execution unit 146. The vector execution unit 146 is, for example, a general-purpose parallel processing unit and may be designed to perform various operations, such as scaling, offsetting, and substitution operations. Thus, unit 146 can be fully utilized to perform the desired accumulation, and therefore, an accumulation circuit at the IMC level can be omitted.

[0052] For completeness, further calculations may need to be performed in the IMC unit in the bit-serial implementation, as will be discussed later.

[0053] 1.1.3.4. Preferred Architecture of IMC Unit Figure 4 shows a programming circuit 158, designed to be sufficiently independent of the computing circuit 15 so that it can proactively reprogram currently inactive weights while the computing circuit is performing MAC operations based on currently active weights. This independence allows for proactive loading of weights required for the next cycle of the operation. Prefetching operations can be performed, for example, for several sets of weights at a time (q≧2). Various prefetching schemes can be considered.

[0054] The programming circuit 158 ​​may, for example, connect the memory unit 11 to the cells of the IMC unit 15. For example, a digital memory cell (in other words, a cell containing a digital memory element) may be connected to dedicated lines that, for example, embody the word line and bit line for write operations in an SRAM memory device. Contrary to what the drawing in Figure 4 suggests, it should be noted that the selection circuit 159 may, in some cases, reuse the word line and bit line for read operations. As a result, the selection circuit and the programming circuit may actually partially overlap.

[0055] In the examples of Figures 4 and 5, it is assumed that each of the N × M memory systems 157 contains K distinct memory elements. Each memory element is adapted to store its respective weight value. In this case, the selection circuit may include N × M multiplexers 159. Each multiplexer is connected to all memory elements of each memory system 157, as in Figure 4. In addition, a selection control line is connected to each multiplexer to allow any of the K weights of each memory system 157 to be selected and set as the active weight during operation. A selection bit may be transmitted through the control line to select the active weight, as illustrated in Figure 5.

[0056] In the example in Figure 5, the multiplexer is a channel multiplexer that uses an inverter and a logic "NAND" gate to reach a common output X. In other words, the combinational logic circuit switches one of several input lines A, B, C, D to a single common output line X. Data lines A, B, C, D are represented by W in Figure 2. 1,1,0 , W 1,1,1 , W 1,1,2 , W 1,1,3The data selection lines (carrying the binary input address) are defined by Add0 and Add1, corresponding to the least significant bit (LSB) and most significant bit (MSB), respectively. For simplicity, a single multiplexer 157 is shown in Figure 4. However, N × M multiplexers are used to switch the weights of N × M memory systems. In principle, to enable individual control of each multiplexer, there are at most 2 × N × M control lines, in other words, 2 control lines for every multiplexer 159. However, in practice, control lines can be shared across multiplexers, and possibly across all multiplexers, especially when it is desired to select weight sets simultaneously. This is because all control lines are preferably shared, which allows the same index k to be selected simultaneously for every element in the M × N memory systems. In such cases, the number of control lines can be reduced to Log2(K).

[0057] Similarly, the programming circuit 158 ​​may be accompanied by N × M demultiplexers, where the same control bit line is used for the entire array 15. For simplicity, Figure 4 shows a single demultiplexer 158 connected to each memory system 157. However, in reality, there are N × M demultiplexers 158 and N × M multiplexers 159 connected to each memory system 157. In a variant, the programming circuit 158 ​​and selection circuit 159 may include other types of electronic components, where such components are located in or connected to at least each cell to program and select memory values ​​as needed. In a further variant, each memory system 157 is configured to store K distinct values ​​at its respective local address, rather than being composed of K distinct memory elements.

[0058] As mentioned above, the required weights are preferably all activated at once. To that end, the selection circuit 159 may be advantageously configured to select a subset consisting of (at least) n × m weights from one of K sets of N × M weights. This is most efficiently achieved by simultaneously selecting the k-th weight from the K weights of each memory system in a subset of the n × m memory systems 157 (where 2 ≤ n ≤ N, 2 ≤ m ≤ M, and 1 ≤ k ≤ K). Activating the weights of n × m subarrays may be advantageous in the case of matrix-vector calculations where not all N × M weights must be switched, which depends on how the problem is initially mapped onto the N × M cells 155. Note that the switching operation may, in rare cases, have to be performed for a single cell (in other words, n=1 and m=1). However, in practice, especially when large operand matrices are involved, weight selection is often performed simultaneously for a large subset of the N×M memory systems (in other words, n>1 and m>1), or even for all N×M memory systems. Nevertheless, in a variant form, the selection circuit 159 can systematically select a set of N×M weights from one of the K sets of N×M weights by simultaneously selecting the k-th weight out of the K weights for each of the N×M memory systems in order to systematically switch all memory systems 157 at the same time. Thus, generally, the selection circuit 159 is configured to select a set of n×m weights for the n×m memory systems of array 15 and set the selected set of weights as the active weights (where 1≦n≦N and 1≦m≦M).

[0059] 1.1.3.5. Preferred Implementation Forms of Vectors and Main Processing Units The following discusses preferred implementations of the VPU 14. As previously mentioned, the information processing unit 10 generally includes a controller 149 for controlling operations involving the IMC unit 15. This controller 149 may, in some cases, form part of the IMC unit, as assumed in Figure 1. As further seen in Figure 1, the VPU 14 may also include a vector instruction decoder 142 connected to the controller 149. In this case, the vector instruction decoder 142 may be configured to decode vector instructions and send the decoded vector instructions to the controller 149. The controller 149 may be used, among other things, to control vector feeding operations, MVM, and vector reading operations. This is achieved thanks to the decoded vector instructions sent from the vector instruction decoder 142. However, in practice, not all vector instructions are necessarily sent to the IMC controller 149. In other words, vector instructions may be sent to other units of the VPU 14, as will be discussed later. The controller 149 may, if necessary, control weight feeding operations based on the decoded instructions sent by the vector instruction decoder 142.

[0060] Vector instructions are used to control and properly synchronize IMC operations to ensure that, at least, the output vector is written to the vector register 148 in a timely manner, provided the register size allows it. For example, the output vector can only be written to the vector register 148 after the IMC unit 15 has properly completed the corresponding MVM and it has been confirmed that the previous output vector has been properly transferred to memory 11 or the main processor 12. In other words, a vector register should not be overwritten by an IMC output unless its contents are no longer needed. However, such synchronization constraints are determined by the size of the vector register. Furthermore, another aspect of synchronization is that the IMC cannot start a calculation cycle until the intended input vector is actually available in the vector register.

[0061] If the controller 149 has direct access to memory 11 (see dashed arrow in Figure 1), the controller 149 may also receive instructions to directly fetch weights from memory 11 (but not vector components). In a variant, weights are transported through vector register 148, as previously described. However, in either case, the corresponding operation can be controlled thanks to vector instructions. Such vector instructions preferably form part of a vector instruction set designed according to a base instruction set architecture. Interestingly, the base instruction set architecture can be extended with two types of vector instructions designed to cause weight storage and MVM execution, respectively. That is, the operations of the IMC unit 15 can be advantageously controlled by extending the existing standardized vector instruction set architecture (ISA) with two new instructions. The first additional instruction is intended to load a data array into the IMC unit, while the second additional instruction is for performing matrix-vector multiplication of the input vector with one of the internal matrices of the IMC unit.

[0062] The following illustrates possible assembly formats for these instructions. First, instructions for storing a vector in a row of a weight matrix can generally be written as "vmvmtx...md.r,vs", where the instruction "vmvmtx" moves the source vector vs from the vector register file to row r of the IMC unit's internal destination matrix md. For example, the instruction "vmvmtx...m0.3,v4" causes the source vector register v4 to move to row 3 of the internal matrix m0. Similarly, instructions for multiplying a matrix by a vector can be written as "vmulmtx...vd,ms,vs", which multiplies the source vector vs from the vector register file by the internal matrix ms and writes the result to the destination vector vd in the vector register file. For example, "vmulmtx...v7,m2,v1" causes the source vector register v1 to be multiplied by the internal matrix m2 and writes the resulting vector to the destination vector register v7.

[0063] It should be noted that the above example of assembly instruction format is consistent with the format used in the RISC-V ISA. That is, the above instructions can be considered a possible extension of the RISC-V ISA. Several variations may be conceivable. For example, the first additional instruction may be modified to further specify whether to load directly from memory (which may be done by the vector load and storage unit 144 or an additional dedicated unit) or from a vector register, depending on the selected data transmission method. This would allow the first additional instruction to specify different memory / register addresses.

[0064] In the example shown in Figure 1, the MPU 12 forms part of the device 10. The MPU 12 interfaces with the VPU 14 to forward vector instructions to the vector instruction decoder 142. However, other architectures are conceivable. Furthermore, in this example, memory 11 also forms part of the device 10. During operation, this memory 11 acts as the main memory for the MPU 12. Memory 11 interfaces with both the MPU 12 and the VPU 14. The MPU 12 further comprises an instruction decoder 122 configured to fetch instructions from memory 11, decode the fetched instructions, and forward the vector instructions to the VPU 14 accordingly.

[0065] The VPU 14 may include a unit 144, referred to in Figure 1 as the “vector load and storage unit,” which is configured to perform vector transfer operations, (if necessary) weight transfer operations, and vector read operations by transferring corresponding values ​​between memory 11 and vector registers 148. During operation, the instruction decoder 122 decodes instructions from memory 11 to identify the type of instruction and which units 12, 14 are responsible for executing them, and thus distinguishes between the conventional data path through execution unit 126 and the vector data path through the VPU 14.

[0066] In addition, the MPU12 generally includes registers 128 (for storing their respective values), a load and storage unit 124 (for transferring data between memory 11 and registers 128), and an arithmetic and logic unit 126. Unit 126 is a processing element that reads values ​​from source registers, performs operations on such values, and stores the results in destination registers among registers 128.

[0067] For completeness, the VPU14 may also include a vector execution unit 146, as previously mentioned. The vector execution unit 146 can, among other things, read vector data from the vector register 148 and, if necessary, perform operations on the corresponding vector, such as element-wise operations on individual vector components, reduction operations, or substitutions of individual vector components. In addition, the vector execution unit 146 stores the output vector in the vector register 148 and, if necessary, accumulates partial values, as discussed earlier. It should be noted that in practice, vector instructions generally distinguish between the source vector register and the destination vector register in register 148. However, both types of instructions relate to the same physical register.

[0068] 1.2. Information Processing Systems The present invention can further be embodied as an information processing system 1, as depicted in Figure 6. In this example, system 1 is a network of interconnected machines, including the three devices 10 described above. More generally, however, system 1 may comprise any number of such devices 10. The devices 10 may, among other things, be connected to a server 2, as further assumed in Figure 6. As a whole, system 1 allows a user to interact with server 2 to accelerate machine learning computation tasks or other tasks involving MVM that are offloaded to the devices 10, as in the embodiment. That is, server 2 interacts with a client 4, which may be a natural person, process, or machine (interacting via a personal computer 3). Each device 10, in this example, is configured to read data from and write data to the memory units of server 2. Client requests are managed by server 2, which may be configured to map a given computing task to, for example, vectors and weights, which are then passed to the devices 10 to efficiently perform the MVM.

[0069] System 1 may be configured as a configurable, disassembled infrastructure that may further include other hardware acceleration devices, such as application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs). More generally, other architectures may be envisioned. For example, System 1 may be configured as a standalone system, possibly connected to one or more general-purpose computers. System 1 may be used in distributed computing systems, such as edge computing systems, among other things.

[0070] 1.3. How to operate the IMC unit using vector processing 1.3.1. Main features and variations Next, the final aspect of the present invention will be described with reference to Figure 7. This aspect relates to a method for operating the IMC unit 15 to perform MVM. As previously described with reference to the first aspect of the present invention, the IMC unit 15 has a crossbar array structure that can store matrix weights and is connected to the vector register 148 of the VPU 14. This method essentially focuses on performing vector transfer operations, vector supply operations, MVM, and vector read operations, as previously discussed with reference to the first aspect.

[0071] The vector transfer operation causes the input vector components of the input vector to be stored in the vector register 148 (see step S144 in Figure 7). Next, the vector supply operation is performed in S145 to supply the input vector components from the vector register as input to the IMC unit 15. The IMC unit 15 is then operated to perform MVM in S146 based on the weights stored across the cells 155 of the crossbar array structure and the input vector components supplied to the IMC unit 15. This causes the output vector to be formed. Finally, the vector read operation is performed in S147 to write the output vector components of the output vector to the vector register 148. From this point onward, a new cycle may be performed based on new input vector components.

[0072] The vector components can be fetched from adjacent memory 11 (interfaced with VPU 14), for example, as discussed earlier, and possibly transferred through vector register 148. Conversely, the output vector components can finally be transferred in S148 from vector register 148 to memory 11 or the connected MPU 12 after the vector read operation is performed in S147.

[0073] The weights are stored in the IMC unit 15 before the MVM operation begins. As mentioned above, such weights may need to be updated. For this purpose, the weights can be fetched (possibly proactively) from memory 11 and then loaded directly into the IMC unit 15. Alternatively, the weights can be transferred through the vector register 148. In this case, the method performs a weight transfer operation in S141 to store the weights in the vector register 148 so that the weights are stored across the cells 155 of the crossbar array structure, and then performs a weight supply operation in S142 to supply the weights to the IMC unit 15. The weight transfer and weight supply operations may have to be performed in stages, depending on the memory capacity of the vector register 148.

[0074] The crossbar array structure includes N × M cells 155, each having a memory system 157. As described in Section 1.1.3.2, the memory system 157 may be advantageously designed to store K weights (K ≥ 2), thereby allowing the N × M memory systems to store K sets of N × M weights. In this case, the N × M weights must be activated as active weights in S149 before the MVM is performed in S146. This is achieved by selecting a weight from its K weights for each memory system 157 and setting the selected weight as the active weight. As previously described, this allows the IMC unit 15 to proactively prefetch weights while performing the MVM based on the currently active weights. In other words, q sets of N×M weights can be prefetched in S141, and these prefetched weights can be stored in N×M memory systems 157 in place of the q sets of N×M weights that are not currently active as active weights (where 1 ≤ q ≤ K-1).

[0075] Various operations can be controlled, for example, thanks to well-designed vector instructions sent in S126 from the MPU12 interfaced with the VPU14. The VPU14 can then decode the vector instructions and execute the decoded instructions to control all necessary operations, including vector supply operations, vector read operations, and weight-related operations if necessary, starting with vector transfer operations. The vector instructions may favorably form part of a vector instruction set, designed according to the underlying ISA and extended with two types of vector instructions, the two types of vector instructions designed to cause weight storage and MVM execution, respectively.

[0076] The embodiments described above are briefly explained with reference to the accompanying drawings and can be adapted to several variations. Several combinations of the above features can be considered. Examples are given in the following sections.

[0077] 1.3.2. Preferred workflow Figure 7 illustrates the preferred flow of operation. Essentially, the MPU 14 (for example, the main core) interacts with memory 11 to load instructions in S122 and decode such instructions in S124. Some of these instructions are processed internally by the MPU, while others are forwarded in S126 to the VPU 12 (referred to as the vector core in Figure 7), which then transfers the weights and input vectors to the IMC unit 15 for the purpose of performing the MVM.

[0078] Upon receiving such an instruction, the VPU 14 dispatches the instruction to the controller 149, which may fetch (or prefetch) the weight data from memory 11 in S141 and then write the weight data across cell 155 of the IMC unit 15 in S142. In a variant, the VPU may first store the weight data in the vector register before feeding the weight data from the vector register to the IMC unit in S142. Loading the weight data may have to be done in stages (S142a: no). The MVM cycle can begin once all the weights necessary to start this cycle are stored in the IMC unit (S142a: yes). Prefetching the weight data (S141, S142) can begin as soon as the MVM cycle is complete (assuming the corresponding weight matrix is ​​no longer needed). The prefetching mechanism is implemented continuously, in other words, in the background, while the IMC unit 15 is performing the MVM.

[0079] A given set of weights is activated in S149 before the MVM begins. For this purpose, the following input vector is selected in S143, the components of this vector are stored in the vector register in S144, and then passed to the IMC unit, which performs the MVM in S146 based on this input vector and the weights currently activated as active weights. The resulting output vector is written to the vector register in S147. The results of successive MVM operations are monitored (S147a: yes, no). When the current MVM is complete (S147a: yes), the output vector is transferred to memory 11 in S148. The transfer is monitored to ensure that the output vector is not overwritten (S148a). Once it is confirmed that the transfer is complete (S148a: yes), the process checks whether all input vectors have been processed (in other words, with respect to the current weight matrix). Otherwise (S148b: No), the following input vector is selected in S143 for the purpose of performing a new MVM operation.

[0080] Note that the flow in Figure 7 ensures that new vector components cannot enter the vector register in S144 before the output vector from the previous MVM operation is transferred to memory 11 (S148a: yes). Once all input vectors for the current weight matrix have been processed (S148b: yes), a new set of weights is activated in S149 to start a new MVM computation cycle. If necessary, additional weights may be proactively fetched in S142 while the MVM is running. Note that the output vector may be accumulated in the vector register in some cases to accommodate larger operands if necessary, as discussed previously. In that case, only the final output vector is transferred to memory 11.

[0081] 2. Details of Particularly Preferred Embodiments and Technical Implementations 2.1 Preferred Architecture of the Device As illustrated in Figure 1, the device 10 preferably includes a memory 11, an MPU 12, and a VPU 14, where the VPU 14 includes an IMC unit 15 integrated into the VPU 12. The VPU is co-integrated with the memory 11 and the MPU 15 on the same chip. In other words, the device 10 is a single device consisting of integrated components 11, 12, 14 (and 15).

[0082] The MPU 12 ("Main Core" in Figure 1) includes an instruction decoder 122 connected to memory 11, which receives and decodes instructions and forwards them to the load and storage unit 124 or the execution unit 126. The load and storage unit 124 exchanges data with memory 11 according to the instructions received from the MPU's execution unit 126 or VPU 12. Both the load and storage unit 124 and the execution unit 126 can write to and read from the register file 128.

[0083] The VPU14 ("Vector Core") includes a vector instruction decoder 142 that interfaces with the MPU12. The vector instruction decoder 142 receives instructions from the instruction decoder 122, decodes such instructions, and forwards the decoded instructions to the relevant entities 144, 146, and 149 of the VPU14. In particular, the vector instruction decoder 142 communicates with the vector load and storage unit 144, which reads input vectors from memory 11 and writes output vectors to memory 11. The vector instruction decoder 142 further communicates with the vector execution unit 146, which, if necessary, performs required operations (e.g., accumulation, substitution, scaling, offset, etc.) on the vectors stored in the vector registers. Finally, the vector instruction decoder 142 communicates with the controller 149 of the IMC unit 15 to properly control all operations related to MVM (in other words, fetching weight data, loading input vectors from vector register 148, executing MVM, and writing output vectors to vector register 148). Note that the controller 149 can fetch weight data directly from memory, as indicated by the dashed arrow, when instructed to do so by the vector instruction decoder 142.

[0084] 2.2 Preferred Architecture for Crossbar Array Structure The crossbar array structure 15 includes N input lines 152 and M output lines 153. The input and output lines 152, 153 are interconnected at intersections (i.e., junctions), which define N × M cells 155. Each cell 155 contains its own memory system 157. Each of the N × M memory systems contains K memory elements, each adapted to store one of K weights. Thus, each memory system 157 can store a maximum of K weights, where K is equal to 4. In total, the crossbar array structure 15 can store a total of K × N × M weights. The input lines 152 and output lines 153 form a 512 × 512 array. Other crossbar array dimensions can be contemplated, as illustrated in Section 1.

[0085] The IMC unit may further comprise an input unit (not shown) for applying input signals to N input lines 152, a programming circuit 158, a selection circuit 159, and a read unit (not shown) which may include an accumulator unless accumulation is performed directly in a vector register. The selection circuit 159 and the input unit may be controlled, for example, by the same logic unit which forms part of the same configuration and control logic circuit and is connected to or forms part of the controller 149.

[0086] The memory element is preferably a digital memory element, such as an SRAM device. More generally, this IMC unit is compatible with various types of electronic memory devices, including non-volatile memory elements (e.g., flash cells). Any type of memristic device may be intended, such as phase-change memory cells, RRAM elements, and electrochemical random-access memory (ECRAM) devices. In other variations, the memory element is an analog memory element. In that case, each multiplication-accumulation operation, in other words, Σ i W i,j,k x iHowever, this is done analogously, and the output signal is converted to the digital domain using an analog-to-digital converter (ADC) circuit.

[0087] For example, each of the K memory elements is a digital memory element, such as an SRAM device. In this case, each cell 155 includes an arithmetic unit 156 (including a multiplier and adder tree, see Figures 3 and 4) connected to each of the K memory elements of each memory system 157 via its respective selection circuit section 159 (e.g., a multiplexer). Each cell is physically connected to each memory element via a selection circuit component (such as a multiplexer or any other selection circuit component), but logically connected to only one such element at a time by the selection made by the selection circuit.

[0088] In the bit-serial implementation, each memory element is designed to store P bit weights. An input unit (not shown) is configured to apply an input signal to the input line 152 to supply components of an N vector bit-serially over P cycles (where P≧1, or more likely P≧2, e.g., P=8 or 16). P corresponds to the bit width of each of the N components of the vector used in the input, and each vector component corresponds to a P bit input word. In this case, N×M cells 155 must be designed to perform MAC operations in bit-serial mode (in other words, over P cycles). The hardware device 10 includes, for example, an accumulator circuit at the output of the column to accumulate values ​​corresponding to the partial bit-serial product obtained in each of the P cycles. Meanwhile, the selection circuit 159 maintains the same set of N×M weights as active weights during each of the P cycles.

[0089] Besides the bit-serial implementation, this concept can also be implemented using a parallel implementation, which does not require a parallel-to-serial conversion. In the parallel implementation, each N vector is processed using weight multiplication in a single cycle. Further variations may involve hybrid methods with parallel supply of bit-serial values.

[0090] Although the present invention has been described in relation to a limited number of embodiments, variations, and accompanying drawings, it will be understood by those skilled in the art that various modifications may be made and replaced with equivalents without departing from the scope of the invention. In particular, features (device-like or method-like) exhibited in a given embodiment or variation, or shown in a certain drawing, may be combined with or replaced with other features in another embodiment, variation, or drawing without departing from the scope of the invention. Thus, various combinations of features described in relation to any of the above embodiments or variations may be contemplated, remaining within the scope of the appended claims. In addition, many minor modifications may be made to adapt the teachings of the invention to specific situations or materials without departing from the scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed, and is intended to include all embodiments that fall within the scope of the appended claims. In addition, many variations other than those explicitly mentioned above may be contemplated. For example, other types of memory elements, selection circuits, and programming circuits may be contemplated.

Claims

1. Information processing device (10), A vector processing unit (14) having a vector register (148) designed to store vector components, An in-memory computing unit (15) or IMC unit having a crossbar array structure, connected to the vector register (148) and adapted to store weights, Equipped with, The aforementioned information processing device (10) A vector transfer operation for storing the input vector component of the input vector in the aforementioned vector register (148), A vector supply operation for supplying the input vector components from the vector register as input to the IMC unit (15) in order for the IMC unit to perform matrix-vector multiplication or MVM based on the weights and the input vector components to obtain an output vector, and Vector read operation for writing the output vector component of the output vector to the vector register (148) An information processing device (10) configured to perform the following.

2. The aforementioned information processing device (10) A weight transfer operation for storing the weights in the vector register (148), and Weight supply operation to supply the weights from the vector register to the IMC unit (15) so that the weights are stored across the crossbar array structure. Further configured to implement, The information processing device (10) according to claim 1, wherein the device (10) is preferably configured to perform the weight transfer operation and the weight supply operation in steps according to the memory capacity and / or throughput capacity of the vector register.

3. The aforementioned crossbar array structure is N × M cells (155) each having a memory system (157) designed to store K weights (K ≥ 2), thereby the crossbar array structure (15) includes N × M cells (155) each containing N × M memory systems (157) adapted to store K sets of N × M weights, A selection circuit (159) is configured to enable N × M weights as active weights (S149) by selecting a weight from its K weights for each of the memory systems (157) and setting the selected weight as an active weight. An information processing apparatus (10) according to claim 1 or 2, including the following:

4. The aforementioned information processing device (10) While the IMC unit (15) performs MVM based on the N × M weights that are currently active, it prefetches q sets of the N × M weights and stores the prefetched weights in the N × M memory systems (157) in place of the q sets of N × M weights that are not currently enabled as active weights (where 1 ≤ q ≤ K - 1). The information processing apparatus (10) according to claim 3, further configured as follows.

5. The aforementioned selection circuit is N x M multiplexers (159), each connected to each of the K memory elements of the N x M memory systems (157), and During operation, a selection control line connected to each of the multiplexers allows any one of the K weights in each of the memory systems (157) to be selected and set as the active weight. The information processing apparatus (10) according to claim 4, including the following:

6. The information processing device (10) further comprises an accumulation circuit interfaced with the vector register (148), The information processing apparatus (10) according to any one of claims 1 to 5, wherein the accumulation circuit is configured to accumulate the output vector components of the output vector acquired from the IMC unit during operation.

7. The information processing device (10) according to any one of claims 1 to 6, wherein the vector register (148) is designed to store a given number of numerical values, the given number being greater than or equal to the smallest value of N and M.

8. The information processing device (10) preferably further includes a controller (149) that forms part of the IMC unit, The vector processing unit (14) includes a vector instruction decoder (142) connected to the controller (149), The vector instruction decoder (142) is configured to decode a vector instruction and transmit the decoded vector instruction to the controller (149), The information processing apparatus (10) according to any one of claims 1 to 7, wherein the controller (149) is configured to control the vector supply operation, the MVM, and the vector read operation based on the decoded vector instruction transmitted from the vector instruction decoder.

9. The information processing device (10) is further configured to store the weights across the crossbar array structure, The aforementioned vector instructions form part of a vector instruction set designed according to a base instruction set architecture extended with two types of vector instructions. The information processing apparatus (10) according to claim 8, wherein the two types of vector instructions are designed to cause the weights to be stored and the MVM to be performed, respectively.

10. The aforementioned information processing device (10) A main processing unit (12) configured to execute non-vector instructions, wherein the main processing unit (12) is interfaced with a vector processing unit (14) to forward the vector instructions to a vector instruction decoder (142). The information processing apparatus (10) according to claim 8 or 9, further comprising:

11. The information processing device (10) further comprises a memory (11) interfaced with the main processing unit (12) and the vector processing unit (14), The main processing unit (12) further comprises an instruction decoder (122) configured to fetch an instruction from the memory (11), decode the fetched instruction, and forward the vector instruction to the vector processing unit (14) accordingly. The information processing apparatus (10) according to claim 10, further comprising a vector load and storage unit (144) configured to perform the vector transfer operation and the vector read operation by transferring corresponding values ​​between the memory (11) and the vector register (148).

12. The IMC unit (15) is integrated into the vector processing unit (14), An information processing device (10) according to any one of claims 1 to 11.

13. An information processing system (1) comprising one or more information processing devices (10) according to any one of claims 1 to 12.

14. A method for operating an in-memory computing unit (15) or IMC unit to perform matrix-vector multiplication or MVM, wherein the IMC unit (15) has a crossbar array structure for storing weights and is connected to a vector register (148) of a vector processing unit (14), and the method is A vector transfer operation is performed to store the input vector component of the input vector in the vector register (148) (S144), A vector supply operation is performed to supply the input vector component from the vector register as input to the IMC unit (15) (S145), Based on the stored weights and the input vector components supplied to the IMC unit (15), the IMC unit (15) is operated to perform matrix-vector multiplication or MVM (S146) to obtain an output vector, A vector read operation is performed to write the output vector component of the output vector to the vector register (148) (S147). Methods that include...

15. Before the method performs the vector transfer operation and the vector supply operation, A weight transfer operation for storing the weights in the vector register (148) (S141), and Weight supply operation (S142) to supply the weights to the IMC unit (15) so that the weights are stored across the crossbar array structure. The method according to claim 14, further comprising carrying out the following.

16. The crossbar array structure comprises N × M cells (155), each having a memory system (157) designed to store K weights (K ≥ 2), thereby comprising N × M cells (155), each containing N × M memory systems adapted to store K sets of N × M weights. The method according to claim 14 or 15, further comprising, before performing the MVM (S146), selecting a weight from its K weights for each of the memory systems (157) and setting the selected weight as an active weight, thereby enabling the N × M weights as active weights (S149).

17. The aforementioned method, While the IMC unit (15) performs MVM based on the currently active weights, it prefetches q sets of N×M weights (S141) and stores the prefetched weights in the N×M memory system (157) in place of the q sets of N×M weights that are not currently enabled as active weights (where 1 ≤ q ≤ K-1). The method according to claim 16, further comprising:

18. The method described above is In the vector processing unit (14), in order to control the weight transfer operation, the weight supply operation, the vector transfer operation, the vector supply operation, and the vector reading operation, a vector instruction is decoded and the decoded instruction is executed. The method according to any one of claims 14 to 17, further comprising:

19. The method according to claim 18, wherein the vector instructions form part of a vector instruction set designed according to a base instruction set architecture extended with two types of vector instructions, and the two types of vector instructions are designed to cause the weight supply operation and the MVM to perform, respectively.

20. The method according to claim 19, further comprising sending the vector instruction from a main processing unit (12) interfaced with the vector processing unit (14) before the vector instruction is decoded in the vector processing unit (14) (S126).

21. The method described above is Before performing the vector transfer operation to store the input vector components of the input vector in the vector register (148), the input vector components are fetched from the memory (11) interfaced with the vector processing unit (14), After performing the vector read operation (S147), the output vector component is transferred from the vector register (148) to the memory (11) (S148). The method according to any one of claims 14 to 20, further comprising:

Images