Systolic array, processing circuit including systolic array, electronic device, and method of operating systolic array

Abstract

A systolic array, a processing circuit including the systolic array, an electronic device, and a method of operating the systolic array are provided. The systolic array includes: a plurality of sub-processing element (PE) arrays, where each of the plurality of sub-PE arrays is configured to receive an input signal and to generate an output signal based on the received input signal; and a plurality of output direct path (ODP) circuits configured to operate in a first mode and in a second mode, where an ODP circuit of the plurality of ODP circuits receives an output signal of a previous one of the plurality of sub-PE arrays, where the ODP circuit provides the output signal to a next one of the plurality of sub-PE arrays in the first mode, and where the ODP circuit provides the output signal to the next one of the plurality of sub-PE arrays in the second mode. And wherein the ODP circuit outputs the output signal in the second mode.

Description

[0001] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2024-0018395 filed on February 6, 2024, in the Korean Intellectual Property Office, and Korean Patent Application No. 10-2024-0052396 filed on April 18, 2024, in the Korean Intellectual Property Office, the contents of which are incorporated herein in their entirety by reference. Technical Field

[0002] The present disclosure relates to processing artificial neural networks (ANNs), and more particularly to systolic arrays (eg, general purpose systolic arrays), processing circuits including systolic arrays, and electronic devices including processing circuits including systolic arrays (eg, general purpose systolic arrays). Background Art

[0003] ANN models can be used to process rich and complex data in a variety of fields including computer vision, speech recognition, natural language processing, video analysis, medical imaging and diagnosis, stock market prediction, autonomous driving, robot navigation, etc. Processing ANNs can involve a large amount of computation, which may require additional consideration of computational efficiency.

[0004] Different aspects of computational efficiency may be more relevant to different applications. For example, power optimization may be an issue when implementing ANNs on a device due to limited battery life. Furthermore, computational speed may be more important for autonomous driving purposes, as short latency may be crucial for ensuring rapid responses to events on the road. Therefore, there is a need in the art for systems and methods that more efficiently process ANNs with respect to both computational speed and latency. Summary of the Invention

[0005] This disclosure describes systems and methods for implementing neural networks (NNs). Embodiments of the disclosure include performing matrix multiplication operations for processing, such as training of NNs (e.g., deep neural networks, convolutional neural networks, etc.). In some cases, a systolic array that can be used to perform matrix multiplication includes an array of processing elements (PEs) that each perform a multiply and accumulate (MAC) operation. The systolic array can operate in different modes based on the density or sparsity of the input matrix, resulting in efficient matrix multiplication operations and increased processing speed.

[0006] The present disclosure provides a general systolic array that operates in two modes based on sparsity in the input matrix (ie, sparsity corresponding to the frequency of zero values) to perform matrix multiplication quickly and efficiently.

[0007] According to one aspect of the present disclosure, a systolic array is provided, comprising: a plurality of sub-processing element (PE) arrays, wherein each of the plurality of sub-PE arrays is configured to receive an input signal and generate an output signal based on the received input signal; and a plurality of output direct path (ODP) circuits configured to operate in a first mode and a second mode, wherein an ODP circuit of the plurality of ODP circuits receives an output signal of a previous sub-PE array of the plurality of sub-PE arrays, wherein the ODP circuit provides the output signal to a next sub-PE array of the plurality of sub-PE arrays in the first mode, and wherein the ODP circuit outputs the output signal in the second mode.

[0008] According to another aspect of the inventive concept, there is provided a processing circuit including a systolic array, wherein the systolic array includes: a plurality of sub-processing element (PE) arrays, wherein each of the plurality of sub-PE arrays is configured to receive an input signal and generate an output signal based on the received input signal; and a plurality of output direct path (ODP) circuits configured to operate in a first mode and a second mode, wherein an ODP circuit of the plurality of ODP circuits receives an output signal of a previous sub-PE array of the plurality of sub-PE arrays, wherein the ODP circuit provides the output signal to a next sub-PE array of the plurality of sub-PE arrays in the first mode, and wherein the ODP circuit outputs the output signal in the second mode.

[0009] According to another aspect of the inventive concept, an electronic device is provided, the electronic device comprising: a processing circuit including a systolic array, wherein the systolic array comprises: a plurality of sub-processing element (PE) arrays, wherein each of the plurality of sub-PE arrays is configured to receive an input signal and generate an output signal based on the received input signal; and a plurality of output direct path (ODP) circuits configured to operate in a first mode and in a second mode, wherein an ODP circuit of the plurality of ODP circuits receives an output signal of a previous sub-PE array of the plurality of sub-PE arrays, wherein the ODP circuit provides the output signal to a next sub-PE array of the plurality of sub-PE arrays in the first mode, and wherein the ODP circuit outputs the output signal in the second mode.

[0010] According to another aspect of the inventive concept, an operating method for a systolic array is provided, the operating method including: receiving, by the systolic array, input data, weight data, and a control signal, wherein the control signal indicates a first mode or a second mode, and the weight data includes uncompressed weight data in the first mode and includes compressed weight data in the second mode; generating, by the systolic array, output data based on the input data, the weight data, and the control signal; and outputting the output data based on the control signal, wherein the output data is provided to an output buffer in the first mode and to a recovery circuit in the second mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the inventive concept will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0012] Figure 1 is a block diagram illustrating a processing circuit including a systolic array according to an embodiment.

[0013] Figure 2 is a block diagram illustrating a processing circuit according to an embodiment.

[0014] Figure 3 is a block diagram illustrating a processing systolic array according to an embodiment.

[0015] Figure 4 is a diagram illustrating the operation of a plurality of sub-processing element (PE) arrays according to an embodiment.

[0016] Figure 5 is a diagram illustrating the operation of one sub-PE array according to an embodiment.

[0017] Figure 6 is a diagram illustrating the operation of one PE according to an embodiment.

[0018] Figure 7 is a diagram illustrating the operation of an output direct path (ODP) circuit according to an embodiment.

[0019] Figure 8 is a diagram illustrating the operation of one ODP circuit according to the embodiment.

[0020] Figure 9 and Figure 10 It shows the operation Figure 3 An illustration of an example method for a systolic array is shown in .

[0021] Figure 11 is a diagram illustrating the operation of a compression circuit according to an embodiment.

[0022] Figure 12 and Figure 13 is a diagram illustrating an example operation of a processing circuit according to an embodiment.

[0023] Figure 14 A system to which the processing circuit according to the embodiment is applied is shown.

[0024] Figure 15 A flow chart describing a process for operating a systolic array according to an embodiment is shown. DETAILED DESCRIPTION

[0025] Machine learning models include systolic array architectures designed for high-performance computing tasks. In some cases, a systolic array can represent a network of processors that rhythmically compute and pass data through the system, effectively parallelizing tasks and improving computational efficiency. With the increasing demand for real-time processing and the exponential growth of data generated by modern electronic devices, there is a need for specialized hardware that can efficiently handle these workloads.

[0026] Existing processing architectures can struggle to keep up with the computational demands of contemporary applications, such as deep learning algorithms and large-scale data analytics. Such systems can suffer from inefficiencies associated with data movement and parallel processing, which lead to bottlenecks that limit performance and increase power consumption. Furthermore, existing systems can lack the flexibility and scalability required to adapt to the diverse and evolving needs of different computing tasks, leading to inefficiencies and a lack of general applicability across a variety of domains.

[0027] The present disclosure describes systems and methods for implementing neural networks (NNs). Matrix multiplication can be used in the field of artificial intelligence for operations such as those in training convolutional neural networks (CNNs) or deep neural networks (DNNs). A systolic array, a structure for performing matrix multiplication, may include an array of processing elements (PEs) that perform multiplication and accumulation (MAC) operations. Embodiments of the present disclosure include performing matrix multiplication operations for processes such as training NNs (e.g., DNNs, etc.). In some cases, a systolic array that can be used to perform multiplication includes an array of processing elements (PEs) that perform multiplication and accumulation (MAC) operations. The systolic array can operate in different modes based on the density or sparsity of the input matrix, resulting in efficient matrix multiplication and increased processing speed.

[0028] Embodiments of the present disclosure include a systolic array comprising a plurality of processing element (PE) subarrays and an output direct path (ODP) circuit. The processing circuitry comprising the systolic array receives an input matrix and generates an output matrix based on a weight matrix by implementing a MAC operation. In some cases, the processing circuitry includes a control circuit configured to generate a first control signal or a second control signal indicating a first mode or a second mode of operation, respectively. For example, the control circuitry may output the first control signal or the second control signal based on a density level of the input matrix.

[0029] In some cases, the systolic array may process an input matrix (i.e., perform a MAC operation on the input matrix) based on the weight matrix in a first mode or based on the compressed weight matrix in a second mode to generate an output matrix. According to an embodiment, in the case of the first mode, the systolic array may be configured to generate first output data (e.g., an output matrix including the first portion of data) and send it to an output buffer of the processing circuit for post-processing or reuse as an input matrix. According to an embodiment, in the case of the second mode, the systolic array may be configured to generate second output data (e.g., an output matrix including the first portion of data and different from the first output data) and send it to a recovery circuit of the processing circuit.

[0030] According to embodiments, the systolic array may operate in a first mode in response to a case where the input matrix is a dense matrix, or may operate in a second mode in response to a case where the input matrix is a sparse matrix, based on sparsity corresponding to the frequency of 0 values of the input matrix. In some cases, in the second mode, the systolic array may include a preloaded weight matrix corresponding to values in the compressed weight matrix.

[0031] In some cases, the compression circuit can be used to compress the input weight matrix. For example, the compression circuit can be configured to divide the input weight matrix into a plurality of sub-row matrices. In some cases, the compression circuit can delete columns containing only zeros in each sub-row matrix to generate a compressed weight matrix for implementing a MAC operation in the second mode. Additionally, the compression circuit can generate metadata to store relevant column information, and can provide the generated metadata to the recovery circuit to recover the output matrix based on the rearrangement of the partial data based on the received metadata.

[0032] This disclosure describes a systolic array including multiple sub-processing element (PE) arrays and multiple output direct path (ODP) circuits. In some cases, each of the multiple sub-PE arrays is configured to receive an input signal and generate an output signal based on the received input signal. Additionally, the multiple ODP circuits are configured to operate in a first mode and a second mode. For example, the ODP circuit may receive an output signal from a previous sub-PE array in the multiple sub-PE arrays, provide the output signal to a next sub-PE array in the first mode, and output the output signal in the second mode. In some examples, the output signal output in the second mode may be different from the output signal in the first mode.

[0033] Embodiments of the present disclosure describe methods for operating a systolic array. In some cases, the systolic array receives input data, weight data, and a control signal. In some cases, the control signal indicates a first mode or a second mode. In some cases, the weight data includes uncompressed weight data in the first mode and compressed weight data in the second mode. The systolic array generates output data based on the input data, the weight data, and the control signal. Additionally, the systolic array outputs the output data based on the control signal. In some cases, the output data is provided to an output buffer in the first mode and to a recovery circuit in the second mode.

[0034] Therefore, by implementing the first and second modes for operating the systolic array, embodiments of the present disclosure can perform efficient matrix multiplication both when the input matrix is a dense matrix and when the input matrix is a sparse matrix. Additionally, by preloading weights corresponding to the values included in the compressed weight matrix into the systolic array in the second mode, embodiments can partially skip operations on zero values, resulting in an improvement in overall processing speed. In the case of the second mode, each of the multiple sub-PE arrays can operate as a systolic array, resulting in a reduction in the number of clock cycles required for MAC operations and a further improvement in overall processing speed.

[0035] The following detailed structural or functional description is provided only as an example, and various changes and modifications may be made to the embodiments. Here, the embodiments are not interpreted as being limited to the present disclosure, and should be understood to include all changes, equivalents, and substitutes within the concept and technical scope of the present disclosure.

[0036] Terms such as first, second, etc. may be used herein to describe components. Each of these terms is not intended to define the nature, order, or sequence of the corresponding component, but is only used to distinguish the corresponding component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0037] It should be noted that if a first component is described as being “connected,” “coupled,” or “engaged” to a second component, although the first component may be directly connected, coupled, or engaged to the second component, a third component may be “connected,” “coupled,” or “engaged” between the first and second components.

[0038] As used herein, unless the context clearly indicates otherwise, the singular also includes the plural. It will also be understood that when the terms "comprises" and / or "comprising" are used herein, they specify the presence of the recited features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0039] As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B or C" may include any one of the items listed together in a corresponding phrase, or all possible combinations thereof.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. Unless expressly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and should not be interpreted in an idealized or overly formal manner.

[0041] Hereinafter, a systolic array, a processing circuit including a systolic array, and an electronic device according to an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. When describing the embodiments with reference to the accompanying drawings, like reference numerals denote like elements, and repeated description thereof will be omitted.

[0042] Figure 1 is a block diagram illustrating a processing circuit including a systolic array according to an embodiment. Hereinafter, the processing circuit including the systolic array 100 will be referred to as a processing circuit 1. Additionally, according to an embodiment, the processing circuit 1 may be referred to as an accelerator.

[0043] Reference Figure 1 , the processing circuit 1 may receive an input matrix and a weight matrix as inputs, and perform a multiplication and accumulation (MAC) operation on the input matrix and the weight matrix to generate an output matrix.

[0044] A machine learning model or artificial neural network (ANN) can be a hardware or software component consisting of connected nodes (i.e., artificial neurons) that loosely correspond to neurons in the human brain. Each connection, or edge, sends a signal from one node to another (like a physical synapse in the brain). When a node receives a signal, it processes it and then sends the processed signal to other connected nodes. An ANN has many parameters, including weights and biases, associated with each neuron in the network. These parameters control the degree of connectivity between neurons and influence the neural network's ability to capture complex patterns in data. These parameters (also called model parameters or model weights) are variables that determine the behavior and characteristics of a machine learning model.

[0045] In some cases, the signals between nodes consist of real numbers, and the output of each node is calculated as a function of its inputs. For example, nodes may use other mathematical algorithms to determine their outputs, such as selecting the maximum value from the inputs as the output, or any other suitable algorithm for activating a node. Each node and edge is associated with one or more node weights that determine how the signal is processed and sent. In some cases, nodes have a threshold below which no signal is sent at all. In some examples, nodes are aggregated into layers.

[0046] The parameters of a machine learning model can be organized into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the final layer is called the output layer. In some cases, the signal passes through a particular layer multiple times. Hidden (or intermediate) layers include hidden nodes and are located between the input and output layers. Hidden layers perform nonlinear transformations on the inputs entering the network. Each hidden layer is trained to produce a defined output that contributes to the combined output of the ANN's output layer. A hidden representation is a machine-readable data representation of the input learned from the ANN's hidden layers and produced by the output layer. As the ANN's understanding of the input improves as the ANN is trained, the hidden representation gradually differentiates from earlier iterations.

[0047] In some cases, ANNs can be processed based on a multiplication and accumulation (MAC) operation, which can be a mathematical process used to calculate the output of neurons in a layer. Each neuron in a neural network receives an input signal, which can be a numerical value representing a feature or activation from the previous layer. The MAC operation can be responsible for combining the input with the corresponding weights associated with each connection. The MAC operation includes two main steps (i.e., multiplication and accumulation).

[0048] For each input to a neuron, multiply the input value by its corresponding weight. Mathematically, if a neuron receives a value with a corresponding weight Input , then the multiplication step generates a series of products: After multiplying each input by its weight, the results are added together. In some cases, "sum" can mean the weighted sum of the inputs. Mathematically, the accumulation step produces a single value: ,in, is the output of the MAC operation.

[0049] According to embodiments of the present disclosure, processing circuit 1 may store the values of the weight matrix in a systolic array. In some cases, processing circuit 1 may even reuse the values of the weight matrix after a MAC operation. In some cases, processing circuit 1 may receive an input matrix (e.g., only the input matrix) as input and perform a MAC operation on the values of the received input matrix based on the weight matrix stored in the systolic array. Thus, processing circuit 1 may generate an output matrix as described herein. Additionally, storing the values of the weight matrix in the systolic array may mean preloading the values of the weight matrix into the systolic array.

[0050] As described, a MAC operation may represent an operation that multiplies two input values and accumulates their results. The MAC operation is an operation used in machine learning (such as neural networks) and signal processing algorithms, and more specifically, is widely used in neural network structures (such as convolutional neural networks (CNNs)).

[0051] Multiplication, as used in MAC operations, may represent a multiplication operation between a weight parameter of a neural network and an input value (e.g., input data). Additionally, accumulation, as used in MAC operations, may represent an operation that calculates a final output value based on the accumulation result of the multiplication between the weight parameter and the input value.

[0052] The number of MAC operations can be used to measure the complexity and computational cost of a neural network model. Additionally, the number of MAC operations of a model can enable evaluation of the model's computational complexity and graph complexity. In some cases, the number of MAC operations can be used as an indicator for optimizing or comparing models. Specifically, the number of MAC operations can enable evaluation of model or hardware performance, or can be used to improve the size and efficiency of a model.

[0053] According to the present disclosure, processing circuit 1 may operate in a first mode when the input matrix is a dense matrix based on sparsity corresponding to the frequency of 0 values of the input matrix. In some cases, processing circuit 1 may operate in a second mode when the input matrix is a sparse matrix.

[0054] Hereinafter, a first mode may indicate a mode in which a dense matrix (i.e., a dense matrix having a frequency of 0 values lower than a predefined frequency) is input as an input matrix. Hereinafter, a second mode may indicate a mode in which a sparse matrix (i.e., a sparse matrix having a frequency of 0 values greater than or equal to a predefined frequency) is input as an input matrix.

[0055] Therefore, the processing circuit 1 of the present disclosure can perform efficient matrix multiplication when the input matrix is a dense matrix and when the input matrix is a sparse matrix.

[0056] According to an embodiment, a systolic array 100 included in a processing circuit 1 includes a plurality of sub-processing element (PE) arrays 110 and a plurality of output direct path (ODP) circuits 120. In some cases, each of the plurality of sub-PE arrays 110 may include m×n PEs arranged in an m×n matrix. In some cases, each of the plurality of ODP circuits 120 may receive an output signal of a previous sub-PE array and provide the output signal of the previous sub-PE array to a next sub-PE array. Additionally or alternatively, each of the plurality of ODP circuits 120 may output the output signal of the previous sub-PE array. As used herein, m and n may represent positive integers. In some cases, the plurality of ODP circuits 120 may each provide the output signal of the previous sub-PE array to the next sub-PE array in a first mode. In some cases, the plurality of ODP circuits 120 may each output the output signal of the previous sub-PE array in a second mode. Figures 2 to 13 Further details regarding the operation of systolic array 100 are described.

[0057] According to an embodiment of the present disclosure, each of the plurality of sub-PE arrays 110 can be individually operated as a systolic array in the second mode. Thus, the number of clock cycles required for MAC operations can be reduced, and the overall processing speed can be improved.

[0058] Figure 2 is a block diagram illustrating a processing circuit according to an embodiment.

[0059] Reference Figure 2 , processing circuit 1 may include a systolic array 100, an input circuit 10, a weight input circuit 20, an output buffer 30, a compression circuit 40, a restoration circuit 50, and a control circuit 60. In some cases, the output buffer 30 and the restoration circuit 50 may be implemented as one output circuit. In some cases, the compression circuit 40 and the restoration circuit 50 may be implemented as one circuit. In some cases, the output buffer 30, the compression circuit 40, and the restoration circuit 50 may be implemented as one circuit.

[0060] Reference Figure 2Input circuit 10 may receive an input matrix as input and output input data as output. In some cases, input circuit 10 performs a preprocessing operation on the input matrix to generate the input data. In some cases, input circuit 10 may be referred to as a scratch pad.

[0061] According to an embodiment, the input circuit 10 may store an input matrix applied externally. In some cases, the input circuit 10 may be implemented in a memory. For example, the input circuit 10 may be implemented in a volatile memory such as a dynamic random access memory (DRAM), mobile DRAM, static random access memory (SRAM), phase change random access memory (PRAM), ferroelectric random access memory (FRAM), resistive random access memory (RRAM), and / or magnetoresistive random access memory (MRAM).

[0062] In some cases, input circuit 10 may generate input data by dividing an input matrix (e.g., dividing the input matrix column by column) and sequentially provide the generated input data to systolic array 100 in synchronization with a clock signal. For example, a piece of input data may correspond to a value included in each column of the input matrix. In some examples, at least one piece of input data may be provided to systolic array 100 simultaneously.

[0063] Additionally, input data corresponding to the first row of the first column of the input matrix and input data corresponding to the first row of the second column of the input matrix may be provided to the systolic array 100 at different points in time. For example, after the input data corresponding to the first row of the first column of the input matrix is provided to the systolic array 100, the input data corresponding to the first row of the second column of the input matrix may be provided to the systolic array 100. In some examples, the input data corresponding to the second row of the first column of the input matrix and the input data corresponding to the first row of the second column of the input matrix may be provided to the systolic array 100 simultaneously.

[0064] The weight input circuit 20 may receive a matrix as an input value and output weight data WDATA as an output value. As described herein, the weight data WDATA may be generated when the weight input circuit 20 performs a preprocessing operation on a matrix (e.g., a weight matrix WEIGHT MATRIX or a compressed weight matrix WEIGHT MATRIX_C).

[0065] In some cases, weight input circuit 20 may receive a weight matrix as input and output weight data WDATA as output in a first mode. Additionally, weight input circuit 20 may receive a compressed weight matrix WEIGHT MATRIX_C as input and output weight data WDATA as output in a second mode.

[0066] Depending on the embodiment, the weight input circuit 20 may store a weight matrix or a compressed weight matrix WEIGHT MATRIX_C (e.g., the compressed weight matrix may be provided externally). In some cases, the weight input circuit 20 may be implemented in a memory. For example, the weight input circuit 20 may be implemented in a volatile memory (such as DRAM, mobile DRAM, SRAM, PRAM, FRAM, RRAM, and / or MRAM).

[0067] The weight input circuit 20 may generate weight data WDATA corresponding to each weight value included in the received matrix and may provide the weight data WDATA to the systolic array 100. As described herein, when performing a MAC operation on the input matrix, a plurality of PEs included in the systolic array 100 may each store the corresponding weight value and may reuse the weight data WDATA.

[0068] In some cases, the weight input circuit 20 may provide weight data WDATA corresponding to the values included in the weight matrix to the systolic array 100 in the first mode (e.g., preload the weight data WDATA corresponding to the values included in the weight matrix to the sub-PE array). In some cases, a plurality of PEs included in the systolic array 100 may store corresponding values among the values included in the weight matrix. When performing a MAC operation on the input matrix, the plurality of PEs may each reuse a corresponding value among the values included in the weight matrix.

[0069] In some cases, the weight input circuit 20 may provide weight data WDATA corresponding to the values included in the compressed weight matrix WEIGHT MATRIX_C to the systolic array 100 in the second mode (e.g., preload the weight data WDATA corresponding to the values included in the compressed weight matrix WEIGHT MATRIX_C to the sub-PE array). In some cases, a plurality of PEs included in the systolic array 100 may each store a corresponding value among the values included in the compressed weight matrix WEIGHT MATRIX_C. During a MAC operation performed on an input matrix, the plurality of PEs may each reuse a corresponding value among the values included in the compressed weight matrix WEIGHT MATRIX_C.

[0070] Therefore, by preloading weights corresponding to values included in the compressed weight matrix WEIGHT MATRIX_C to the systolic array 100 in the second mode, embodiments of the present disclosure may be able to partially skip operations on 0 values, resulting in an improvement in the overall processing speed of the system.

[0071] The output buffer 30 may receive the first output data as input and generate an output matrix as output in the first mode. In some cases, when the output buffer 30 rearranges the first output data, an output matrix may be obtained.

[0072] Additionally, the output buffer 30 may store the first output data or the output matrix. Depending on the embodiment, the output buffer 30 may be implemented using a memory. For example, the output buffer 30 may be implemented using a volatile memory (such as DRAM, mobile DRAM, SRAM, PRAM, FRAM, RRAM, and / or MRAM).

[0073] According to an embodiment, output buffer 30 may perform functional operations corresponding to post-processing (such as activation, normalization, and / or max pooling) according to an artificial intelligence (AI) algorithm. Additionally, output buffer 30 may store the result of post-processing performed on the first output data (i.e., the output matrix) in output buffer 30. In some cases, output buffer 30 may apply this result to processing circuit 1 (e.g., input circuit 10 of processing circuit 1) as input to processing circuit 1 (i.e., output buffer 30 may enable the output matrix to be reused as input to processing circuit 1). Therefore, input circuit 10 and output buffer 30 may be bidirectionally connected.

[0074] Compression circuitry 40 may receive a matrix (e.g., a weight matrix) as input and generate a compressed matrix as output. In some cases, the compressed matrix may be obtained when compression circuitry 40 performs a matrix compression operation on the weight matrix. As described herein, a matrix compression operation may refer to an operation that reduces the dimension of a matrix.

[0075] Reference Figure 2 , compression circuit 40 may receive a weight matrix as input and generate a compressed weight matrix WEIGHTMATRIX_C as output. In some cases, when compression circuit 40 performs a matrix compression operation on the received weight matrix, a compressed weight matrix WEIGHTMATRIX_C may be obtained.

[0076] Additionally, the compression circuit 40 may generate compressed metadata METADATA_C by generating a compressed matrix, and provide the compressed metadata METADATA_C to the restoration circuit 50 .

[0077] According to an embodiment, the compression circuit 40 may generate compressed metadata METADATA_C by generating a compressed weight matrix WEIGHT MATRIX_C, and provide the compressed metadata METADATA_C to the restoration circuit 50 .

[0078] In some cases, the compression metadata METADATA_C may represent metadata corresponding to a compressed matrix generated using a matrix compression operation. For example, the compression metadata METADATA_C may include information about the characteristics of the weight matrix, the dimensions and structure of the weight matrix, and the algorithm or parameters used during the matrix compression operation.

[0079] The restoration circuit 50 may receive the second output data as input and generate an output matrix as output in the second mode. In some cases, the restoration circuit 50 may generate a partial sum matrix based on the second output data. Additionally, the restoration circuit 50 may generate restored partial sum matrices corresponding to the generated partial sum matrices by rearranging the generated partial sum matrices based on the compressed metadata METADATA_C. In some cases, the restoration circuit 50 may generate an output matrix based on the sum of the restored partial sum matrices.

[0080] The compression circuit 40 and / or the restoration circuit 50 may be implemented in software, hardware, or a combination of hardware and software.

[0081] Will refer to Figures 11 to 13 Further details regarding the operation of compression circuit 40 and restoration circuit 50 are described.

[0082] The control circuit 60 may control various operations of the processing circuit 1. For example, the processing circuit 1 may operate in a first mode in response to a first control signal CON1 of the control circuit 60, and may operate in a second mode in response to a second control signal CON2 of the control circuit 60.

[0083] According to an embodiment, the control circuit 60 may output a first control signal CON1 in response to an external command instructing operation in the first mode. In some cases, the control circuit 60 may output a second control signal CON2 in response to an external command instructing operation in the second mode. The processing circuit 1 may operate in the first mode in response to the first control signal CON1, or operate in the second mode in response to the second control signal CON2.

[0084] According to some embodiments, the control circuit 60 may generate the first control signal CON1 or the second control signal CON2 based on the sparsity corresponding to the frequency of zero values included in the input matrix. For example, when the frequency of zero values included in the input matrix is less than a preset value, the control circuit 60 may output the first control signal CON1. Therefore, the processing circuit 1 may operate in the first mode in response to the first control signal CON1. For example, when the frequency of zero values included in the input matrix is greater than or equal to a preset value, the control circuit 60 may output the second control signal CON2. Therefore, the processing circuit 1 may operate in the second mode in response to the second control signal CON2.

[0085] Systolic array 100 may include an array of PEs that perform MAC operations. In some cases, the PEs may form a connection network and may perform MAC operations synchronously with a global clock signal.

[0086] Additionally, the systolic array 100 may include three data streams (ie, an input static data stream, a weight static data stream, and an output static data stream) according to values stored in the systolic array 100 .

[0087] In the case of weight static data flow, weights are preloaded into each PE in the systolic array 100 and are not moved from each PE when performing a MAC operation.

[0088] According to an exemplary embodiment, the data flow of the systolic array 100 is assumed to be a weight-static data flow in which weights are stored in the systolic array 100 .

[0089] However, embodiments are not limited to systolic arrays 100 with static weights and may be applied to systolic arrays with input static data flows and systolic arrays with output static data flows.

[0090] According to an embodiment, the systolic array 100 may receive input data as input and generate first output data as output in a first mode. Additionally, the systolic array 100 may receive input data as input and generate second output data as output in a second mode. In some cases, the first output data may be the result of a MAC operation based on a weight matrix. Unlike the first output data, the second output data may be the result of a MAC operation based on a compressed weight matrix WEIGHT MATRIX_C.

[0091] Will refer to Figures 3 to 13 Further details regarding the operation of systolic array 100 are described.

[0092] Figure 3 A systolic array according to an embodiment is shown. Figure 4 The operation of multiple sub-PE arrays according to an embodiment is shown.

[0093] The systolic array 100 may include a plurality of sub-PE arrays 110 and a plurality of ODP circuits 120 .

[0094] As used herein, the plurality of sub-PE arrays 110 may each include m×n PEs arranged in an m×n matrix. Here, m and n may be positive integers.

[0095] Additionally, each of the plurality of ODP circuits 120 may receive an output signal of a previous sub-PE array in a first mode and provide the output signal of the previous sub-PE array to a next sub-PE array. Additionally or alternatively, each of the plurality of ODP circuits 120 may receive an output signal of a previous sub-PE array in a second mode and output the output signal of the previous PE array. For example, each of the plurality of ODP circuits 120 may provide the output signal of the previous sub-PE array to the recovery circuit 50 or provide the output signal to an external device in the second mode.

[0096] Reference Figure 3 , the systolic array 100 may include a first sub-PE array 110_1 to a K-th sub-PE array 110_K and a first ODP circuit 120_1 to a (K-1)th ODP circuit 120_K-1. Figure 3 As shown, the number of ODP circuits may be one less than the number of sub-PE arrays. In some cases, K may be a positive integer.

[0097] Will refer to Figures 4 to 6 Further details regarding the operation of the multiple sub-PE array 110 are provided.

[0098] Figure 4 The multiplication of an input matrix and a weight matrix according to an embodiment is shown.

[0099] Reference Figure 3 and Figure 4 , the input matrix can be divided into at least one column. For example, the input matrix can be divided into K sub-column matrices (i.e., Figure 4 , including the first sub-column matrix A1 to the K-th sub-column matrix Ak).

[0100] Additionally, each sub-column matrix may correspond to one of the multiple input data. For example, Figure 4 Each of the first sub-column matrix A1 to the K-th sub-column matrix Ak may correspond to Figure 3 For example, the first sub-column matrix A1 may correspond to the first input data INPUT DATA_1, and the second sub-column matrix A2 may correspond to the second input data INPUT DATA_2.

[0101] Additionally, the weight matrix may be divided into at least one row. For example, the weight matrix may be divided into K sub-row matrices (ie, Figure 4 The first sub-row matrix W1 to the K-th sub-row matrix Wk).

[0102] In some cases, each sub-row matrix may correspond to weight data stored in multiple sub-PE arrays. For example, Figure 4 Each of the first sub-row matrix W1 to the K-th sub-row matrix Wk can be stored in Figure 3 The weight matrix is stored in one of the first to Kth sub-PE arrays 110_1 to 110_K. For example, the first sub-row matrix W1 may be stored in the first sub-PE array 110_1, and the second sub-row matrix W2 may be stored in the second sub-PE array 110_2. In some cases, storing the sub-row matrix of the weight matrix in the sub-PE array may be referred to as preloading the weights into the sub-PE array.

[0103] Refer again Figure 3 Each sub-PE array in the plurality of sub-PE arrays 110 may receive corresponding input data and a result of a MAC operation of a previous sub-PE array as input, and each sub-PE array in the plurality of sub-PE arrays 110 may provide the result of the MAC operation to a corresponding ODP circuit.

[0104] For example, first through K-th sub-PE arrays 110_1 through 110_K may receive as input the first through K-th input data INPUT DATA_1 and the result of a MAC operation performed by the previous sub-PE array (i.e., since first sub-PE array 110_1 is the starting point of the MAC operation, first sub-PE array 110_1 does not receive the result of the previous MAC operation). Each of first through (K-1)th sub-PE arrays 110_1 through 110_K-1 may provide the result of the MAC operation to a corresponding ODP circuit. In some cases, K-th sub-PE array 110_K may provide the result of the MAC operation to the output buffer 30.

[0105] In some cases, the result of the MAC operation may mean an accumulated partial sum representing the result of multiplying the weight value and the input value.

[0106] Additionally, the result of the MAC operation received by the output buffer 30 in the first mode may be the first output data (ie, the first output data may be the final output of the systolic array 100 ).

[0107] In some cases, the output matrix may be generated based on rearranging the first output data. For example, the output matrix may be obtained by multiplying the input matrix by the weight matrix.

[0108] Each of the plurality of ODP circuits 120 may be provided between two sequential sub-PE arrays and may provide an output signal of a previous sub-PE array to a next sub-PE array. In some cases, each ODP circuit may provide an output signal of a previous sub-PE array to the outside of the systolic array 100 according to an operation mode (i.e., a first mode or a second mode). Figure 2As described, the plurality of ODP circuits 120 may each operate in the first mode in response to the first control signal CON1 or in the second mode in response to the second control signal CON2 .

[0109] For example, the first ODP circuit 120_1 may be disposed between the first sub-PE array 110_1 and the second sub-PE array 110_2 .

[0110] In the first mode, each of the plurality of ODP circuits 120 may receive an output signal of a previous sub-PE array as an input and provide the output signal of the previous sub-PE array to a next sub-PE array. For example, in the first mode, the first ODP circuit 120_1 may receive an output signal of the first sub-PE array 110_1 as an input and provide the output signal of the first sub-PE array 110_1 to the second sub-PE array 110_2.

[0111] In the second mode, each of the plurality of ODP circuits 120 may provide an output signal of a previous sub-PE array to the recovery circuit 50. For example, in the second mode, the first ODP circuit 120_1 may receive an output signal of the first sub-PE array 110_1 as an input and provide the output signal of the first sub-PE array 110_1 to the recovery circuit 50.

[0112] In the second mode, one sub-PE array and one ODP circuit corresponding to the sub-PE array may be referred to as a sub-systolic array. Additionally, multiple sub-systolic arrays may each independently operate as a systolic array.

[0113] For example, in the case of the first mode, a start time point (eg, time period) of applying the first input data INPUT DATA_1 to the first sub PE array 110_1 and a start time point (eg, time period) of applying the second input data INPUT DATA_2 to the second sub PE array 110_2 may not overlap with each other.

[0114] In some cases, in the second mode, the first sub-PE array 110_1 and the first ODP circuit 120_1 may constitute a sub-systolic array. Additionally, the second sub-PE array 110_2 and the second ODP circuit 120_2 may constitute a sub-systolic array. Therefore, in the second mode, the start time (e.g., time period) of applying the first input data INPUT DATA_1 to the first sub-PE array 110_1 and the start time (e.g., time period) of applying the second input data INPUT DATA_2 to the second sub-PE array 110_2 may overlap. For example, in the second mode, the time period for receiving the first input data INPUT DATA_1 at the first sub-PE array 110_1 and the time period for receiving the second input data INPUT DATA_2 at the second sub-PE array 110_2 may overlap.

[0115] In some cases, the plurality of sub-systolic arrays may each be individually operated as a single systolic array. Thus, the number of clock cycles required for MAC operations may be reduced and the processing speed of the processing circuit 1 may be increased.

[0116] In the second mode, each of the plurality of ODP circuits 120 may provide the result of the MAC operation of the previous sub-PE array to the recovery circuit 50. In some cases, the result of the MAC operation may mean an accumulated partial sum representing the result of multiplying the weight value and the input value.

[0117] For example, in the case of the second mode, the first ODP circuit 120_1 may provide the first partial sum PARTIAL OUTPUT_1 (i.e., the first partial sum may represent the result of the MAC operation of the first sub-PE array 110_1) to the recovery circuit 50. Additionally, in the case of the second mode, the second ODP circuit 120_2 may provide the second partial sum PARTIAL OUTPUT_2 (i.e., the second partial sum may represent the result of the MAC operation of the second sub-PE array 110_2) to the recovery circuit 50.

[0118] Therefore, in the second mode, the first through (K−1)th ODP circuits 120_1 through 120_K−1 may each provide a partial sum that is a result of a MAC operation from a corresponding sub-PE array to the recovery circuit 50 .

[0119] Additionally, in the second mode, the output buffer 30 may operate as an ODP circuit. For example, in the second mode, the output buffer 30 may provide the K-th partial sum PARTIAL OUTPUT_K (i.e., the K-th partial sum may represent the result of the MAC operation of the K-th sub-PE array 110_K) to the recovery circuit 50.

[0120] Hereinafter, in the second mode, the first to Kth partial sum PARTIAL OUTPUT_1 to PARTIAL OUTPUT_K may be referred to as second output data.

[0121] The restoration circuit 50 may receive as input the second output data including the first to Kth partial sums PARTIAL OUTPUT_1 to PARTIAL OUTPUT_K in the second mode. The restoration circuit 50 may generate a partial sum matrix based on the second output data. Additionally, the restoration circuit 50 may rearrange the generated partial sum matrices based on the compressed metadata METADATA_C to generate restored partial sum matrices corresponding to the generated partial sum matrices. Additionally, the restoration circuit 50 may provide an output matrix by summing each of the restored partial sum matrices.

[0122] In some cases, an output matrix corresponding to the multiplication of the input matrix and the weight matrix can be generated, and the output matrix can be completed by generating the partial sum matrix based on the partial sums (e.g., the first partial sum PARTIAL OUTPUT_1 to the Kth partial sum PARTIAL OUTPUT_K), generating the restored partial sum matrix by rearranging the partial sum matrix, and performing summation of each of the restored partial sum matrices.

[0123] Figure 5 The operation of one sub-PE array according to an embodiment is shown. Figure 6 The operation of one PE according to an embodiment is shown. Figure 5 Show Figure 3 The example of the first sub-PE array 110_1 shown in FIG. Figure 3 The multiple sub-PE arrays 110 described herein may include the same configuration and perform the same operations. Figure 3 The configuration and operation of the first sub-PE array 110_1 are shown in FIG.

[0124] Reference Figure 5 , the first sub-PE array 110_1 may include m×n PEs arranged in an m×n matrix. Here, m and n may be positive integers.

[0125] Figure 6 Describes the operation of PE.

[0126] Figure 6 Show Figure 5 The operation of a PE among the multiple PEs shown in .

[0127] A PE may receive an input value A and a partial sum P as input. In some cases, the PE may provide the result of the MAC operation (A×W+P) to the next PE in the first direction via a path from partial sum P -> A×W+P. The PE may perform a multiplication on the input value A and the preloaded weight W and accumulate the result of the multiplication in the partial sum P, thereby generating the result of the MAC operation (A×W+P).

[0128] Additionally, the PE may provide the input value A to the next processing element PE in the second direction through the path of the input value A.

[0129] In some cases, a PE may have two data paths including a path for input value A and a path for a partial sum P, wherein a first direction may correspond to the path for the partial sum P and a second direction may correspond to the path for input value A. Figure 3 、 Figure 5 and Figure 6 , the sub-PE array can be connected to the ODP circuit on the path of the partial and P in a first direction.

[0130] Reference Figure 4 and Figure 5 , the processing circuit 1 may preload the weight data WDATA corresponding to the first sub-row matrix W1 into the m×n PEs of the first sub-PE array 110_1 arranged in an m×n matrix. In some cases, the dimensions of the first sub-row matrix W1 may be m×n. In some cases, the elements of the first sub-row matrix W1 may be stored in the PEs.

[0131] For example, the PE in the first row and first column may store a weight W11, the PE in the first row and second column may store a weight W12, and the PE in the second row and first column may store a weight W21, and so on.

[0132] Reference Figure 4 and Figure 5 , the first sub-PE array 110_1 may receive as input the first input data INPUT DATA_1 corresponding to the first sub-column matrix A1. In some cases, the input circuit 10 may perform a preprocessing operation on the first sub-column matrix A1 of the input matrix to generate the 1-1th input data INPUT DATA_1-1 through the 1-mth input data INPUT DATA_1-m. In some cases, the dimension of the first sub-column matrix A1 may be t×m, where t may be a positive integer. In some embodiments, the first sub-column matrix A1 may include input data a11 through atm, and the first sub-row matrix W1 may include weights W11 through Wmn.

[0133] Input circuit 10 may synchronize first input data INPUT DATA_1 with a clock signal via a plurality of channels. In some cases, input circuit 10 may provide the synchronized first input data INPUT DATA_1 to first sub-PE array 110_1. In some examples, the number of channels may correspond to the number of columns of first sub-column matrix A1 and / or the number of rows of first sub-row matrix W1.

[0134] For example, the input circuit 10 may synchronize 1-1th to 1-mth input data INPUT DATA_1-1 with a clock signal and provide the synchronized data to the first sub-PE array 110_1 through m channels within m+t-1 clock cycles.

[0135] Additionally, the input circuit 10 can synchronize the elements of a column of the sub-column matrix (e.g., A1) with a clock signal through a channel and sequentially provide the synchronized elements to the PEs of the corresponding row of the first sub-PE array 110_1. Additionally, the elements of a column (i.e., the corresponding column) of the corresponding sub-column matrix (e.g., A1) can be sequentially moved between the PEs of the corresponding row in synchronization with the clock signal.

[0136] Reference Figure 5 For example, the input circuit 10 may synchronize the elements of the first column of the first sub-column matrix A1 with the clock signal via the 1-1th input data INPUT DATA_1-1 channel and sequentially provide the synchronized elements to the first through nth PEs in the first row of the first sub-PE array 110_1. Additionally, the elements of the first column of the first sub-column matrix A1 may be sequentially shifted between the first through nth PEs in the first row of the first sub-PE array 110_1 in synchronization with the clock signal.

[0137] Similarly, if Figure 5 As shown in FIG, the input circuit 10 may synchronize the elements of the m-th column of the first sub-column matrix A1 with the clock signal via the 1-m-th input data INPUT DATA_1-m channel, and may sequentially provide the synchronized elements to the first PE in the m-th row through the n-th PE in the first sub-PE array 110_1. Therefore, the elements of the m-th column of the sub-column matrix (e.g., A1) may be sequentially shifted between the first PE in the m-th row through the n-th PE in the first sub-PE array 110_1 in synchronization with the clock signal.

[0138] Refer again Figure 5The start times of applying the 1-1th to 1-mth input data INPUT DATA_1-1 and INPUT DATA_1-m may be different from each other. For example, after the 1-1th input data INPUT DATA_1-1 is provided to the first sub-PE array 110_1 and one clock cycle has passed, the 1-2th input data INPUT DATA_1-2 may be provided to the first sub-PE array 110_1.

[0139] The first sub-PE array 110_1 may output the first part and PARTIALOUTPUT_1 in synchronization with the clock signal through a plurality of channels. In some cases, the number of channels may correspond to the number of columns of the first sub-row matrix W1.

[0140] Reference Figure 3 , the first sub-PE array 110_1 may provide the first part and PARTIAL OUTPUT_1 to the first ODP circuit 120_1 through a plurality of channels in synchronization with the clock signal.

[0141] Reference Figure 5 , the first through nth PEs in the mth row of the first sub-PE array 110_1 may output PARTIAL OUTPUT_1-1 through PARTIAL OUTPUT_1-n through the channels of the partial sum P. In some cases, the final output of the first PE in the mth row may be output first. Subsequently, the final output of the second PE in the mth row may be output after one clock cycle has passed. Similarly, the final output of the third PE in the mth row may be output after another clock cycle has passed.

[0142] Figure 7 The operation of an ODP circuit according to an embodiment is shown. Figure 7 It shows Figure 3 The block diagram of the operation of the first ODP circuit 120_1 is shown in FIG. Figure 3 The multiple ODP circuits 120 described may include the same configuration and perform the same operation. Figure 3 Further details of the operation of the first ODP circuit 120_1 are shown in FIG.

[0143] Each of the plurality of ODP circuits 120 may receive an output signal of a previous sub-PE array. In some cases, the ODP circuit may provide the output signal of the previous sub-PE array to the next sub-PE array in a first mode, and may provide the output signal of the previous PE array in a second mode. For example, each of the plurality of ODP circuits 120 may provide the output signal of the previous sub-PE array to the recovery circuit 50 or provide the output signal to an external device in the second mode.

[0144] In some cases, each of the plurality of ODP circuits 120 may operate in a first mode in response to a first control signal CON1 or in a second mode in response to a second control signal CON2 .

[0145] Reference Figure 7 The first ODP circuit 120_1 may operate in the first mode in response to the first control signal CON1. In the first mode, the first ODP circuit 120_1 may receive the output signal of the first sub-PE array 110_1 and provide the output signal of the first sub-PE array 110_1 to the second sub-PE array 110_2.

[0146] Additionally, the first ODP circuit 120_1 may operate in a second mode in response to the second control signal CON2 . In the second mode, the first ODP circuit 120_1 may receive an output signal of the first sub-PE array 110_1 and provide the output signal of the first sub-PE array 110_1 to the recovery circuit 50 .

[0147] In some cases, the output signal of the previous sub-PE array can be a partial sum. Figure 7 , the output signals of the first sub-PE array 110_1 may be part 1-1 and PARTIAL OUTPUT_1-1 through part 1-n and PARTIAL OUTPUT_1-n. Therefore, the output signals of the first sub-PE array 110_1 may be the first part and PARTIAL OUTPUT_1. Additionally, the output signals of the second sub-PE array 110_2 may be part 2-1 and PARTIAL OUTPUT_2-1 through part 2-n and PARTIAL OUTPUT_2-n. Therefore, the output signals of the second sub-PE array 110_2 may be the second part and PARTIAL OUTPUT_2.

[0148] Figure 8 The operation of an ODP circuit according to an embodiment is shown. Figure 8 It shows Figure 3 1 is a block diagram of the configuration of the first ODP circuit 120_1 shown in FIG. Figure 3 The multiple ODP circuits 120 described herein may include the same configuration and perform the same operations. Figure 3 Further details of the configuration of the first ODP circuit 120_1 are shown in FIG.

[0149] The plurality of ODP circuits 120 may each include a corresponding sub-output buffer and a plurality of multiplexers.

[0150] Reference Figure 8, the first ODP circuit 120_1 may include a first sub-output buffer 121_1 and 1-1th to 1-nth multiplexers 123_1 - 1 - 123_1 - n .

[0151] The corresponding sub-output buffer may receive an output signal of a previous sub-PE array as an input. Additionally, a plurality of multiplexers may be connected to the corresponding sub-output buffer.

[0152] Reference Figure 8 First sub-output buffer 121_1 may receive the output signal of first sub-PE array 110_1 as input. In some cases, the output signal of first sub-PE array 110_1 may be PARTIAL OUTPUT_1-1 through PARTIAL OUTPUT_1-n. Therefore, the output signal of first sub-PE array 110_1 may be PARTIAL OUTPUT_1.

[0153] Additionally, the corresponding sub-output buffer can provide the output signal of the previous sub-PE array to the multiple multiplexers in the first mode in response to the first control signal CON1. In some cases, the multiple multiplexers can provide the output signal of the previous sub-PE array to the next sub-PE array in the first mode.

[0154] Reference Figure 8 , the first sub-output buffer 121_1 may provide the output signal of the first sub-PE array 110_1 to the 1-1th to 1-nth multiplexers 123_1-1-1 and 123_1-n in the first mode in response to the first control signal CON1.

[0155] Additionally or alternatively, the corresponding sub-output buffers may output the output signal of the previous sub-PE array in the second mode in response to the second control signal CON2. For example, the corresponding sub-output buffers may each provide the output signal of the previous sub-PE array to the recovery circuit 50 in the second mode, or may provide the output signal to the outside.

[0156] Reference Figure 8 In the second mode, the first sub-output buffer 121_1 may provide the output signal of the first sub-PE array 110_1 in response to the second control signal CON2. For example, the first sub-output buffer 121_1 may provide the output signal of the first sub-PE array 110_1 to the recovery circuit 50 in the second mode, or may provide the output signal of the first sub-PE array 110_1 to the outside.

[0157] Each of the plurality of multiplexers may receive as input a corresponding partial sum of a previous sub-PE array and may provide a corresponding partial sum or a zero value to a next sub-PE array in response to a control signal CON1 or CON2.

[0158] Reference Figure 8 , the 1-1th multiplexer 123_1-1 may receive the 1-1th portion and PARTIALOUTPUT_1-1 of the first sub-PE array 110_1 as inputs, and provide the corresponding partial sum or a value of 0 to the second sub-PE array 110_2 in response to the control signal CON1 or CON2. For example, when the first control signal CON1 indicates a bit value of 1 and the second control signal CON2 indicates a bit value of 0, the 1-1th multiplexer 123_1-1 may provide the 1-1th portion and PARTIAL OUTPUT_1-1 to the second sub-PE array 110_2 in response to the first control signal CON1, and may provide a value of 0 to the second sub-PE array 110_2 in response to the second control signal CON2.

[0159] Figure 9 and Figure 10 Show operation Figure 3 An exemplary method of systolic array 100 is shown in FIG. Figure 9 The operation of the systolic array 100 operating in a first mode is shown, and Figure 10 Operation of the systolic array 100 is shown operating in a second mode.

[0160] Reference Figure 9 and Figure 10 The systolic array 100 may include a plurality of sub-PE arrays 110 and a plurality of ODP circuits 120 .

[0161] Reference Figure 9 Further details are provided regarding the operation of the systolic array 100 operating in the first mode.

[0162] Refer to it together Figure 2 and Figure 9 , the systolic array 100 may operate in the first mode in response to the first control signal CON1 of the control circuit 60 .

[0163] In some cases, in the first mode, weight input circuit 20 may receive a weight matrix as input and preload weight data WDATA corresponding to the weight matrix to systolic array 100 .

[0164] In the first mode, each of the plurality of sub-PE arrays 110 may receive corresponding input data and the result of the MAC operation of the previous sub-PE array as input. In some cases, each of the plurality of sub-PE arrays 110 may provide the result of the MAC operation to the corresponding ODP circuit.

[0165] For example, in the first mode, the first through K-th PE arrays 110_1 through 110_K may receive the first through K-th input data INPUT DATA_1 and the result of a MAC operation performed by the previous PE array. As described herein, since the first PE array 110_1 is the starting point of the MAC operation, the first PE array 110_1 does not receive the result of the previous MAC operation as input. Each of the first through (K-1)th PE arrays 110_1 through 110_K-1 may provide the result of the MAC operation to a corresponding ODP circuit, and the K-th PE array 110_K may provide the result of the MAC operation to the output buffer 30.

[0166] In some cases, the result of the MAC operation may represent an accumulated partial sum that is a result of multiplying the weight value and the input value.

[0167] Additionally, the corresponding input data can be the input circuit 10 to the reference Figure 5 Any one of the 1-1th input data INPUT DATA_1-1 to the 1-mth input data INPUTDATA_1-m generated when the pre-processing operation is performed on the corresponding sub-column matrix of the described input matrix.

[0168] The result of the MAC operation received by output buffer 30 in the first mode may represent first output data. In some cases, the first output data may be the final output of systolic array 100.

[0169] Therefore, the output matrix obtained by multiplying the input matrix by the weight matrix can be completed by rearranging the first output data.

[0170] The processing described in the first mode can be performed in synchronization with a clock signal.

[0171] Refer to it together Figure 2 and Figure 10 , the systolic array 100 may operate in the second mode in response to the second control signal CON2 of the control circuit 60 .

[0172] In some cases, in the second mode, compression circuit 40 may receive a weight matrix as input and may provide a compressed weight matrix WEIGHT MATRIX_C to weight input circuit 20. In some cases, compression circuit 40 may generate compressed metadata METADATA_C by generating the compressed matrix and may provide the compressed metadata METADATA_C to restoration circuit 50.

[0173] Additionally, the weight input circuit 20 may receive the compressed weight matrix WEIGHT MATRIX_C as input and may preload weight data WDATA corresponding to the compressed weight matrix WEIGHT MATRIX_C to the systolic array 100 .

[0174] Additionally, in the second mode, each of the plurality of sub-PE arrays 110 may receive corresponding input data and the result of the MAC operation of the previous sub-PE array as input. In some cases, each of the plurality of sub-PE arrays 110 may provide the result of the MAC operation to the corresponding ODP circuit.

[0175] For example, in the second mode, the first through K-th sub-PE arrays 110_1 through 110_K may receive the first through K-th input data INPUT DATA_1 and the result of a MAC operation performed on the previous sub-PE array. As described herein, since the first sub-PE array 110_1 is the starting point of the MAC operation, the first sub-PE array 110_1 may not receive the result of the previous MAC operation as input. Each of the first through (K-1)th sub-PE arrays 110_1 through 110_K-1 may provide the result of the MAC operation to the corresponding ODP circuit, and the K-th sub-PE array 110_K may provide the result of the MAC operation to the output buffer 30.

[0176] In the second mode, each of the plurality of ODP circuits 120 may provide a result of a MAC operation of a previous sub-PE array to the recovery circuit 50. In some cases, the result of the MAC operation may represent an accumulated partial sum as a result of multiplying a weight value by an input value.

[0177] For example, Figure 10 As shown in FIG, in the second mode, the first ODP circuit 120_1 may provide the first partial sum PARTIAL OUTPUT_1 (i.e., PARTIAL OUTPUT_1 may be the result of the MAC operation of the first sub-PE array 110_1) to the recovery circuit 50. Additionally, in the second mode, the second ODP circuit 120_2 may provide the second partial sum PARTIAL OUTPUT_2 (i.e., PARTIAL OUTPUT_2 may be the result of the MAC operation of the second sub-PE array 110_2) to the recovery circuit 50.

[0178] In some cases, in the second mode, each of the first through (K−1)th ODP circuits 120_1 through 120_K−1 may provide a partial sum (ie, the partial sum may represent a result of a MAC operation from a corresponding sub-PE array) to the recovery circuit 50 .

[0179] In some cases, the output buffer 30 may operate as an ODP circuit in the second mode. For example, in the second mode, the output buffer 30 may provide the Kth partial and PARTIAL OUTPUT_K (i.e., PARTIAL OUTPUT_K may be the result of the MAC operation of the Kth sub-PE array 110_K) to the recovery circuit 50.

[0180] In case of the second mode, the first to Kth parts PARTIAL OUTPUT_1 to PARTIAL OUTPUT_K may be referred to as second output data.

[0181] The restoration circuit 50 may receive as input the second output data including the first partial sum PARTIAL OUTPUT_1 to the Kth partial sum PARTIAL OUTPUT_K in the second mode. The restoration circuit 50 may generate a partial sum matrix based on the second output data. Additionally, the restoration circuit 50 may generate restored partial sum matrices corresponding to the generated partial sum matrices. In some cases, the restored partial sum matrix may be generated by rearranging the partial sum matrices based on the compressed metadata METADATA_C. Additionally, the restoration circuit 50 may output an output matrix based on the sum of the restored partial sum matrices. As described herein and with reference to Figure 2 As described, compressed metadata METADATA_C may be provided from the compression circuit 40 .

[0182] Therefore, an output matrix based on the product (i.e., multiplication) of the input matrix and the weight matrix can be generated using a partial sum matrix based on the partial sums (e.g., the first partial sum PARTIAL OUTPUT_1 to the Kth partial sum PARTIAL OUTPUT_K). Subsequently, a restored partial sum matrix can be generated by rearranging the partial sum matrix, and the sum of the restored partial sum matrices can be calculated.

[0183] The processing described in the second mode can be performed in synchronization with the clock signal.

[0184] Additionally, in the second mode, a sub-PE array and an ODP circuit corresponding to the sub-PE array can be referred to as a sub-systolic array. In some cases, multiple sub-systolic arrays can each independently operate as a single systolic array. Thus, multiple sub-systolic arrays can operate in parallel.

[0185] For example, in the first mode, a start time point (eg, time period) of applying the first input data INPUT DATA_1 to the first sub PE array 110_1 and a start time point (eg, time period) of applying the second input data INPUT DATA_2 to the second sub PE array 110_2 may not overlap with each other.

[0186] In some cases, in the second mode, the first sub-PE array 110_1 and the first ODP circuit 120_1 may constitute a sub-systolic array, and the second sub-PE array 110_2 and the second ODP circuit 120_2 may constitute a sub-systolic array. Therefore, in the second mode, the start time point (e.g., time period) of applying the first input data INPUT DATA_1 to the first sub-PE array 110_1 and the start time point (e.g., time period) of applying the second input data INPUT DATA_2 to the second sub-PE array 110_2 may overlap with each other.

[0187] Therefore, each of the plurality of sub-systolic arrays can be individually operated as a systolic array. As a result, the number of clock cycles required for MAC operation can be reduced and the processing speed of the processing circuit 1 can be increased.

[0188] Figure 11 The operation of the compression circuit according to the embodiment is shown. Figure 11 Shown according to the embodiment Figure 2 The operation of the compression circuit 40 is shown in FIG. Figure 11 The method by which the compression circuit 40 generates the compressed weight matrix WEIGHT MATRIX_C is shown.

[0189] Figure 11 A weight matrix of 4×8 dimensions is shown. As shown here, white cells represent elements having a value of 0 among the elements included in the weight matrix WEIGHT MATRIX. Additionally, gray cells represent elements having a value other than 0 among the elements included in the weight matrix WEIGHT MATRIX.

[0190] In the case of the second mode, the compression circuit 40 may perform a compression operation on the received weight matrix in response to the second control signal CON2. In some cases, a compressed weight matrix WEIGHT MATRIX_C may be generated based on the compression operation.

[0191] In some cases, compression circuit 40 may divide the weight matrix into k sub-row matrices, each having the same number (m) of rows. Here, k may correspond to the total number of sub-PE arrays. Additionally, here, m (i.e., the number of rows included in one sub-row matrix) may correspond to the number of PE rows included in one sub-PE array.

[0192] Reference Figure 11 , the compression circuit 40 may divide the weight matrix (eg, the weight matrix in the original dense format having four rows) into two sub-row matrices, wherein each sub-row matrix includes two rows. Figure 11As shown in , the weight matrix may be divided into a plurality of groups. For example, the total number of sub-PE arrays may be two, and the number of rows of PEs included in one sub-PE array may be two.

[0193] Thereafter, compression circuit 40 may delete columns where the value of each element is 0 (e.g., using Figure 11 ). Thus, the compression circuitry may maintain at least one column in each sub-column matrix whose elements have values other than 0 (e.g., a column with at least one gray cell). Thus, a compressed weight matrix WEIGHT MATRIX_C may be generated.

[0194] For example, refer to Figure 11 In the case of the first sub-row matrix, the compression circuit 40 may delete the first, third, sixth, and seventh columns, each of which has a value of 0, and may maintain the 0th, second, fourth, and fifth columns, each of which has an element having a value other than 0. Similarly, in the case of the second sub-row matrix, the compression circuit may delete the 0th, first, third, fifth, and sixth columns, each of which has a value of 0, and may maintain the second, fourth, and seventh columns, each of which has an element having a value other than 0. As a result, a compressed weight matrix WEIGHT MATRIX_C may be generated.

[0195] Additionally, the compression circuit 40 may generate compressed metadata METADATA_C based on the generated compressed matrix, and may provide the compressed metadata METADATA_C to the restoration circuit 50 .

[0196] In some cases, when compression circuitry 40 performs a compression operation, compression metadata METADATA_C may be generated to include the original column indices of the columns maintained in each sub-row matrix.

[0197] Reference Figure 11 , the compression circuit 40 may generate the compressed metadata METADATA_C to include the original column indices 0, 2, 4, 5 of the columns maintained in the first sub-row matrix and the original column indices 2, 4, and 7 of the columns maintained in the second sub-row matrix.

[0198] Figure 12 and Figure 13 An example operation of a processing circuit according to an embodiment is shown. Figure 12 An example of the systolic array 100 operating in the second mode is shown. In some cases, the total number of sub-PE arrays may be assumed to be two, the number of PE rows included in one sub-PE array may be assumed to be two, and the number of ODP circuits may be assumed to be one.

[0199] Figure 12A process is shown that includes a systolic array 100 that performs matrix multiplication based on performing a MAC operation on a 4×4 dimensional weight matrix WEIGHTMATRIX [0 a 0 0; 0 0 0 b; 0 c 0 0; 0 d 0 0] and a 4×4 dimensional input matrix [1 0 0 4; 0 3 0 0; 20 0 5; 0 0 6 0].

[0200] Refer to it together Figure 2 and Figure 12 , the systolic array 100 may operate in the second mode in response to the second control signal CON2 of the control circuit 60 .

[0201] In some cases, in the second mode, compression circuit 40 may receive a weight matrix as input and provide a compressed weight matrix WEIGHT MATRIX_C to weight input circuit 20. As described herein and with reference to Figure 10 As described above, the compression circuit 40 may generate compressed metadata METADATA_C by generating a compressed matrix, and may provide the compressed metadata METADATA_C to the restoration circuit 50. Figure 12 , the compressed metadata METADATA_C may correspond to the matrix [1 3 0 0; 1 0 0 0].

[0202] Additionally, the weight input circuit 20 may receive the compressed weight matrix WEIGHT MATRIX_C as input and may preload weight data WDATA corresponding to the compressed weight matrix WEIGHT MATRIX_C to the systolic array 100 .

[0203] Reference Figure 12 , the compression circuit 40 may perform a compression operation on the input weight matrix (i.e., WEIGHT MATRIX) in the second mode to generate a compressed weight matrix WEIGHT MATRIX_C [a 0 0 0; 0 b 0 0; c 0 0 0; d 0 0 0] and provide the generated compressed weight matrix WEIGHT MATRIX_C [a 0 0 0; 0 b 0 0; c 0 0 0; d 0 0 0] to the weight input circuit 20. Additionally, the compression circuit 40 may generate a matrix [13 0 0; 1 0 0 0] corresponding to the compressed metadata METADATA_C and may provide the compressed metadata METADATA_C to the restoration circuit 50. As described herein and with reference to Figure 11 As described, the compression metadata METADATA_C may include the original index of the column in each sub-row matrix maintained when performing the compression operation.

[0204] Additionally, the weight input circuit 20 may receive the compressed weight matrix WEIGHT MATRIX_C [a 0 0 0; 0 b0 0; c 0 0 0; d 0 0 0] as input and may preload weight data WDATA corresponding to the compressed weight matrix WEIGHT MATRIX_C [a 0 00; 0 b 0 0; c 0 0 0; d 0 0 0] to the systolic array 100 .

[0205] Reference Figure 12 In the second mode, first sub-PE array 110_1 and second sub-PE array 110_2 may receive first input data INPUT DATA_1 and second input data INPUT DATA_2, respectively. Additionally, second sub-PE array 110_2 may receive the result of the MAC operation performed by first sub-PE array 110_1 as input. In some cases, since first sub-PE array 110_1 is the starting point of the MAC operation, first sub-PE array 110_1 may not receive the result of the previous MAC operation.

[0206] The input circuit 10 may generate input data by dividing the input matrix column by column (e.g., based on dividing the input matrix column by column). Additionally, the input circuit 10 may sequentially provide the input data to the systolic array 100 in synchronization with a clock signal. As used herein, a piece of input data may correspond to a value included in each column of the input matrix, and at least one piece of input data may be provided to the systolic array 100 at the same time.

[0207] Additionally, the input matrix may be divided into at least one column. For example, the input matrix may be divided into K sub-column matrices (i.e., referring to Figure 4 1 to the K-th sub-column matrix Ak described above). Additionally, each sub-column matrix may correspond to one of the input data. For example, Figure 4 Each of the first sub-column matrix A1 to the K-th sub-column matrix Ak in may correspond to Figure 3 One of the first to K-th input data INPUT DATA_1 to INPUT DATA_K described in .

[0208] Refer again Figure 12 The input matrix can be divided into two sub-column matrices, each including two columns. The first sub-column matrix A1 (i.e., [1 0; 0 3; 2 0; 0 0]) can correspond to the first input data INPUT DATA_1. Similarly, the second sub-column matrix A2 (i.e., [0 4; 0 0; 0 5; 6 0]) can correspond to the second input data INPUT DATA_2.

[0209] Therefore, the input circuit 10 may provide first input data INPUT DATA_1 corresponding to the first sub-column matrix A1 (i.e., [1 0; 0 3; 2 0; 0 0]) to the first sub-PE array 110_1. Additionally, the input circuit 10 may provide second input data INPUT DATA_2 corresponding to the second sub-column matrix A2 (i.e., [0 4; 0 0; 0 5; 6 0]) to the second sub-PE array 110_2.

[0210] In some cases, the input circuit 10 may synchronize the first input data INPUT DATA_1 with the clock signal through a plurality of channels and may provide the synchronized first input data INPUT DATA_1 to the first sub-PE array 110_1. As used herein, the number of channels may correspond to the reference number. Figure 4 and Figure 5 The number of columns of the first sub-column matrix A1 and / or the number of rows of the first sub-row matrix W1 described above are determined. Additionally, the input circuit 10 can synchronize the elements of a column of the first sub-column matrix A1 with the clock signal via one channel and sequentially provide the synchronized elements to the PEs of the corresponding row of the first sub-PE array 110_1. Additionally, the elements of the corresponding column of the corresponding sub-column matrix (e.g., the first sub-column matrix A1) can be sequentially moved between the PEs of the corresponding row in synchronization with the clock signal.

[0211] Reference Figure 12 The input circuit 10 may synchronize the 1-1th input data INPUT DATA_1-1 and the 1-2th input data INPUT DATA_1-2 with the clock signal and provide the synchronized data to the first sub-PE array 110_1. Additionally, the 1-1th input data INPUT DATA_1-1 may be sequentially shifted between the first to fourth PEs in the first row of the first sub-PE array 110_1 in synchronization with the clock signal.

[0212] Thereafter, the first sub-PE array 110_1 may provide the result of the MAC operation to the first ODP circuit 120_1 in synchronization with the clock signal, and the second sub-PE array 110_2 may provide the result of the MAC operation to the output buffer 30 in synchronization with the clock signal.

[0213] Refer again Figure 12, the first sub-output buffer 121_1 of the first ODP circuit 120_1 may output a matrix (i.e., [a; 0; 2a; 0]) as the first sub-PE array 110_1's 1-1 portion and PARTIAL OUTPUT_1-1. Similarly, the first sub-output buffer 121_1 of the first ODP circuit 120_1 may output a matrix (i.e., [0; 3b; 0; 0]) as the first sub-PE array 110_1's 1-2 portion and PARTIAL OUTPUT_1-2, a matrix (i.e., [0; 0; 0; 0]) as the first sub-PE array 110_1's 1-3 portion and PARTIAL OUTPUT_1-3, and a matrix (i.e., [0; 0; 0; 0]) as the first sub-PE array 110_1's 1-4 portion and PARTIAL OUTPUT_1-4. Additionally, the output buffer 30 may output the matrix [4d; 0; 5d; 6c] (i.e., portion 2-1 of the second sub-PE array 110_2 and PARTIAL OUTPUT_2-1), the matrix [0; 0; 0; 0] (i.e., portion 2-2 of the second sub-PE array 110_2 and PARTIAL OUTPUT_2-2), the matrix [0; 0; 0; 0] (i.e., portion 2-3 of the second sub-PE array 110_2 and PARTIAL OUTPUT_2-3), and the matrix [0; 0; 0; 0] (i.e., portion 2-4 of the second sub-PE array 110_2 and PARTIAL OUTPUT_2-4).

[0214] like Figure 12As shown in , the first sub-output buffer 121_1 and the output buffer 30 can provide the first part and PARTIAL OUTPUT_1 (i.e., the first part and PARTIAL OUTPUT_1 corresponding to the 1-1 part and PARTIAL OUTPUT_1-1 to the 1-4 part and PARTIAL OUTPUT_1-4) and the second part and PARTIAL OUTPUT_2 (i.e., the second part and PARTIAL OUTPUT_2 corresponding to the 2-1 part and PARTIAL OUTPUT_2-1 to the 2-4 part and PARTIAL OUTPUT_2-4) to the recovery circuit 50. In some cases, the first part and PARTIAL OUTPUT_1 (i.e., the first part and PARTIAL OUTPUT_1 corresponding to part 1-1 and PARTIAL OUTPUT_1-1 to part 1-4 and PARTIAL OUTPUT_1-4) and the second part and PARTIAL OUTPUT_2 (i.e., the second part and PARTIAL OUTPUT_2 corresponding to part 2-1 and PARTIALOUTPUT_2-1 to part 2-4 and PARTIAL OUTPUT_2-4) may be referred to as second output data.

[0215] Figure 13 A method of generating an output matrix based on the second output data is shown. In some cases, the recovery circuit 50 can be used to generate the output matrix. Figure 12 The detailed description is continuously provided and repeated descriptions may be omitted.

[0216] The restoration circuit 50 may generate a partial sum matrix based on the second output data. In some cases, the restoration circuit 50 may generate restored partial sum matrices corresponding to the generated partial sum matrices. In some cases, the restoration circuit 50 may generate the restored partial sum matrix by rearranging the generated partial sum matrices based on the compressed metadata METADATA_C. Additionally, the restoration circuit 50 may output an output matrix based on the sum of the restored partial sum matrices. In some cases, the compressed metadata METADATA_C may be provided by the compression circuit 40.

[0217] Reference Figure 13, the recovery circuit 50 can receive the matrix [a; 0; 2a; 0] (i.e., part 1-1 of the first sub-PE array 110_1 and PARTIAL OUTPUT_1-1), the matrix [0; 3b; 0; 0] (i.e., part 1-2 of the first sub-PE array 110_1 and PARTIAL OUTPUT_1-2), the matrix [0; 0; 0; 0] (i.e., part 1-3 of the first sub-PE array 110_1 and PARTIAL OUTPUT_1-3), and the matrix [0; 0; 0; 0] (i.e., part 1-4 of the first sub-PE array 110_1 and PARTIAL OUTPUT_1-4). Additionally, the recovery circuit 50 may receive the matrix [4d; 0; 5d; 6c] (i.e., portion 2-1 of the second sub-PE array 110_2 and PARTIAL OUTPUT_2-1), the matrix [0; 0; 0; 0] (i.e., portion 2-2 of the second sub-PE array 110_2 and PARTIAL OUTPUT_2-2), the matrix [0; 0; 0; 0] (i.e., portion 2-3 of the second sub-PE array 110_2 and PARTIAL OUTPUT_2-3), and the matrix [0; 0; 0; 0] (i.e., portion 2-4 of the second sub-PE array 110_2 and PARTIAL OUTPUT_2-4).

[0218] The restoration circuit 50 may generate a partial sum matrix based on the second output data.

[0219] Reference Figure 13, the restoration circuit 50 may generate the first part and matrix [a 0 0 0; 0 3b 0 0; 2a 0 0 0; 0 0 0 0] by connecting the matrix [a; 0; 2a; 0] (i.e., part 1-1 of the first sub PE array 110_1 and PARTIAL OUTPUT_1-1), the matrix [0; 3b; 0; 0] (i.e., part 1-2 of the first sub PE array 110_1 and PARTIAL OUTPUT_1-2), the matrix [0; 0; 0; 0] (i.e., part 1-3 of the first sub PE array 110_1 and PARTIALOUTPUT_1-3), and the matrix [0; 0; 0; 0] (i.e., part 1-4 of the first sub PE array 110_1 and PARTIAL OUTPUT_1-4). Additionally, the restoration circuit 50 may generate a second portion and matrix [4d 0 0 0; 0 0 0 0; 5d 0 0 0; 6c 0 0 0] by connecting the matrix [4d; 0; 5d; 6c] (i.e., portion 2-1 of the second sub PE array 110_2 and PARTIAL OUTPUT_2-1), the matrix [0; 0; 0; 0] (i.e., portion 2-2 of the second sub PE array 110_2 and PARTIAL OUTPUT_2-2), the matrix [0; 0; 0; 0] (i.e., portion 2-3 of the second sub PE array 110_2 and PARTIAL OUTPUT_2-3), and the matrix [0; 0; 0; 0] (i.e., portion 2-4 of the second sub PE array 110_2 and PARTIAL OUTPUT_2-4).

[0220] Reference Figure 13 , the restoration circuit 50 may generate a restored first partial sum matrix based on the first row element [1 3 0 0] of the compressed metadata METADATA_C corresponding to the matrix [1 3 0 0; 1 0 0 0]. For example, the restoration circuit 50 may shift the 0th column of the generated first partial sum matrix to the first column based on the first column value 1 in the first row element of the metadata METADATA_C. Additionally, the restoration circuit 50 may shift the first column of the generated first partial sum matrix to the third column based on the second column value 3 in the first row element of the metadata METADATA_C. Thus, the restoration circuit 50 may generate a restored first partial sum matrix.

[0221] In some cases, the restoration circuit 50 may generate the restored second partial sum matrix based on the second row element [1 0 0 0] of the compressed metadata METADATA_C corresponding to the matrix [1 3 0 0; 1 0 0 0]. For example, the restoration circuit 50 may generate the restored second partial sum matrix by shifting the 0th column of the generated second partial sum matrix to the first column according to the first column value 1 in the second row element of the metadata METADATA_C.

[0222] Reference Figure 13 , the restoration circuit 50 may output an output matrix based on the sum of the restored first partial sum matrix and the restored second partial sum matrix.

[0223] According to an embodiment of the present disclosure, since the systolic array 100 includes the plurality of ODP circuits 120, the processing circuit 1 may operate in a first mode when the input matrix may be a dense matrix, or may operate in a second mode when the input matrix may be a sparse matrix, based on sparsity corresponding to the frequency of 0 values in the input matrix.

[0224] Therefore, according to the present disclosure, the processing circuit 1 can perform efficient matrix multiplication when the input matrix may be a dense matrix and when the input matrix may be a sparse matrix.

[0225] According to the present disclosure, operations on 0 values can be partially skipped by preloading weights corresponding to values included in the compressed weight matrix WEIGHT MATRIX_C to the systolic array 100 in the second mode. Therefore, the overall processing speed can be improved.

[0226] According to the present disclosure, since the systolic array 100 includes multiple ODP circuits 120, multiple sub-PE arrays can each operate as a systolic array in the second mode (i.e., can operate independently). As a result, the number of clock cycles required for MAC operations can be reduced, thereby improving the overall processing speed.

[0227] Figure 14 A system to which the processing circuit according to the embodiment can be applied is shown.

[0228] Reference Figure 14 ,exist Figure 14 The system 2000 described in may basically be a mobile system (such as a mobile phone, a smartphone, a tablet personal computer (PC), a wearable device, a healthcare device, or an Internet of Things (IoT) device). However, Figure 14 The system 2000 is not necessarily limited to a mobile system and may include a personal computer, a laptop computer, a server, a media player, or an automotive device (such as a navigation device). The system 2000 may also be referred to as an electronic device.

[0229] Reference Figure 14 , the system 2000 may include a main processor 2100, memories 2200a and 2200b, and storage devices 2300a and 2300b, and may additionally include at least one of an image capture device 2410, a user input device 2420, a sensor 2430, a communication device 2440, a display 2450, a speaker 2460, a power supply device 2470, and a connection interface 2480.

[0230] The main processor 2100 may control the overall operation of the system 2000, and more specifically, control operations of other components constituting the system 2000. The main processor 2100 may be implemented by a general-purpose processor, a dedicated processor, or an application processor.

[0231] The main processor 2100 may include one or more CPU cores 2110 and may also include a controller 2120 for controlling memories 2200a and 2200b and / or storage devices 2300a and 2300b. Depending on the embodiment, the main processor 2100 may also include an accelerator 2130, which may represent a dedicated circuit for high-speed data operations (such as AI data operations). The accelerator 2130 may include a graphics processing unit (GPU), a neural processing unit (NPU), and / or a data processing unit (DPU). In some cases, the accelerator 2130 may be implemented as a separate chip that is physically independent of the other components of the main processor 2100.

[0232] In some cases, accelerator 2130 may correspond to a reference Figures 1 to 13 The processing circuit 1 is described.

[0233] The memory 2200a and the memory 2200b may be used as the main memory device of the system 2000 and may include a volatile memory (such as SRAM and / or DRAM). However, embodiments are not limited thereto, and the memory 2200a and the memory 2200b may also include a non-volatile memory (such as flash memory, PRAM and / or RRAM). The memory 2200a and the memory 2200b may be implemented in the same package as the main processor 2100.

[0234] Memory devices 2300a and 2300b can function as nonvolatile memory devices that store data regardless of whether power is supplied to them, and can have relatively large storage capacities compared to memories 2200a and 2200b. Memory devices 2300a and 2300b may include memory controllers 2310a and 2310b, and nonvolatile memories (NVMs) 2320a and 2320b that store data under the control of memory controllers 2310a and 2310b. NVMs 2320a and 2320b may include flash memory having a 2D or 3D V-NAND (vertical NAND) structure, but may also include other types of nonvolatile memory, such as PRAM and / or RRAM.

[0235] The storage devices 2300a and 2300b may be included in the system 2000 but physically separated from the main processor 2100, or may be implemented in the same package as the main processor 2100. Additionally, the storage devices 2300a and 2300b may be solid-state devices (SSDs) or memory cards, and thus the storage devices 2300a and 2300b may be detachably attached to other components of the system 2000 through an interface (such as the connection interface 2480 described below). The storage devices 2300a and 2300b may be devices to which a standard protocol (such as Universal Flash Storage (UFS), embedded MultiMediaCard (eMMC), or Non-Volatile Memory Express (NVMe)) is applied, but are not necessarily limited thereto.

[0236] The image capture device 2410 may capture still images or moving pictures and may include a camera, a video camera, and / or a webcam.

[0237] The user input device 2420 may receive various types of data input from a user of the system 2000 and may include a touch pad, a keypad, a keyboard, a mouse, and / or a microphone.

[0238] The sensor 2430 may sense various types of physical quantities available from outside the system 2000 and may convert the sensed physical quantities into electrical signals. The sensor 2430 may include a temperature sensor, a pressure sensor, an illumination sensor, a position sensor, an acceleration sensor, a biosensor, and / or a gyroscope sensor.

[0239] The communication device 2440 may transmit and receive signals to and from other devices outside the system 2000 according to various communication protocols. The communication device 2440 may include an antenna, a transceiver, and / or a modem.

[0240] Display 2450 and speaker 2460 may be used as output devices for outputting visual information and auditory information, respectively, to a user of system 2000 .

[0241] The power supply device 2470 may appropriately convert power supplied from a battery (not shown) embedded in the system 2000 and / or power supplied from an external power source, and may supply the converted power to the components of the system 2000 .

[0242] The connection interface 2480 may provide a connection between the system 2000 and an external device that can be connected to the system 2000 and exchange data with the system 2000. The connection interface 2480 may be implemented as one of various interface protocols such as Advanced Technology Attachment (ATA), Serial ATA (SATA), external SATA (e-SATA), Small Computer Small Interface (SCSI), Air Attached SCSI (SAS), Peripheral Component Interconnect (PCI), PCI Express (PCIe), NVM Express (NVMe), IEEE 1394, Universal Serial Bus (USB), Secure Digital (SD) card, MultiMediaCard (MMC), eMMC, UFS, embedded Universal Flash (eUFS), and Compact Flash (CF) card interfaces.

[0243] Figure 15 Methods for operating a systolic array according to embodiments of the present disclosure are described. In some examples, these operations are performed by a system including a processor that executes a collection of code to control functional elements of the device. Additionally or alternatively, specialized hardware is used to perform certain processes. Generally, these operations are performed according to the methods and processes described in accordance with aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or performed in conjunction with other operations.

[0244] Embodiments of the present disclosure are configured to perform operations on a systolic array. In some cases, the systolic array may be configured to perform a matrix multiplication operation (e.g., for performing a neural network training operation) on an input matrix to generate an output matrix. For example, the matrix multiplication operation may represent a multiply and accumulate (MAC) operation. Depending on the embodiment, the systolic array may be configured to execute a first mode or a second mode based on receiving a first control signal or a second control signal, respectively. In some examples, the control signal may be based on the density (e.g., or sparsity) of the input matrix.

[0245] According to an embodiment, when the density of the input matrix is higher than a threshold value, the control circuit generates a first control signal. Additionally, according to an embodiment, when the density of the input matrix is less than or equal to the threshold value, the control circuit generates a second control signal. Therefore, by implementing different modes in the systolic array based on the control signal, the embodiment of the present disclosure can perform skipping of zero weight values, which leads to an enhancement in the processing speed of the processing circuit. With reference to at least Figures 2 to 3 、 Figures 7 and 8 and Figures 10 and 11 Further details are provided regarding implementation of the second mode.

[0246] At operation 1510, the system receives input data, weight data, and a control signal. In some cases, the operations of this step represent or may be performed by a systolic array. In some cases, the control signal may be configured to indicate a first mode or a second mode. In some cases, the weight data includes uncompressed weight data in the first mode and compressed weight data in the second mode. Figure 11 Details are provided regarding compression of the matrix by the compression circuit.

[0247] At operation 1520, the system generates output data based on the input data, weight data, and control signals. In some cases, the operations of this step represent or may be performed by a systolic array.

[0248] In some cases, the systolic array generates output data that includes partial data. For example, the systolic array generates output data based on performing a MAC operation on the input data and the weight data. For example, in a first mode, the weight data may represent uncompressed weight data. For example, in a second mode, the weight data may represent compressed weight data (e.g., Figure 11 The generated compressed weight data is shown in ). Figures 2 to 6 Further details on output data generation are provided.

[0249] At operation 1530, the system outputs the output data based on the control signal. In some cases, the operations of this step represent or may be performed by a systolic array.

[0250] In some cases, the systolic array provides output data to an output buffer in a first mode, where the output buffer can use the output matrix for post-processing (e.g., normalization, activation processing in neural network operations). Additionally or alternatively, the output buffer can reuse the output matrix as an input matrix.

[0251] In some cases, the systolic array provides output data to the recovery circuit in the second mode. For example, the systolic array can be configured to send partial data to the recovery circuit. In some examples, the recovery circuit can recover the partial data (e.g., the partial sum matrix generated in the sub-PE array) to generate a sum matrix as the output matrix. Figures 2 to 3 、 Figures 7 and 8 、 Figure 10 and Figures 12 to 13 Further details on the recovery circuit are provided.

[0252] The processes discussed above are intended to be illustrative and not restrictive. Those skilled in the art will understand that the steps of the processes discussed herein may be omitted, modified, combined and / or rearranged, and any additional steps may be performed, without departing from the scope of the invention. More generally, the above disclosure is intended to be exemplary and not restrictive. Only the appended claims are intended to set the boundaries of what is encompassed by the present invention. Furthermore, it should be noted that the features and limitations described in any one embodiment may be applied to any other embodiment herein, and that the flowcharts or examples associated with one embodiment may be combined with any other embodiment in a suitable manner, performed in a different order, or performed in parallel. In addition, the systems and methods described herein may be executed in real time. It should also be noted that the above-described systems and / or methods may be applied to, or used in accordance with, other systems and / or methods.

[0253] While the inventive concept has been particularly shown and described with reference to embodiments thereof, it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. A systolic array comprising: a plurality of sub-processing element arrays, wherein each of the plurality of sub-processing element arrays is configured to receive an input signal and generate an output signal based on the received input signal; and a plurality of output direct path circuits configured to operate in a first mode and a second mode, wherein an output direct path circuit of the plurality of output direct path circuits receives an output signal of a previous sub-processing element array of the plurality of sub-processing element arrays, wherein the output direct path circuit provides an output signal to a next sub-processing element array of the plurality of sub-processing element arrays in the first mode, and wherein the output direct path circuit outputs the output signal in the second mode.

2. The systolic array of claim 1, wherein: Each of the plurality of output direct path circuits is disposed between two consecutive sub-processing element arrays of the plurality of sub-processing element arrays.

3. The systolic array of claim 1 , wherein: A first time period for receiving a first input signal at a first sub-processing element array among the plurality of sub-processing element arrays overlaps with a second time period for receiving a second input signal at a second sub-processing element array among the plurality of sub-processing element arrays.

4. The systolic array of claim 1 , wherein: The plurality of sub-processing element arrays are preloaded with weight values corresponding to the weight matrix in the first mode, and are preloaded with weight values corresponding to the compressed weight matrix in the second mode.

5. The systolic array of claim 1 , wherein: The output direct path circuit outputs an output signal of a previous sub-processing element array to the restoration circuit in the second mode.

6. The systolic array of claim 1 , wherein: The output direct path circuit includes a sub-output buffer and one or more multiplexers, wherein the sub-output buffer is configured to receive an output signal of a previous sub-processing element array as an input, and wherein the one or more multiplexers are each connected to the sub-output buffer.

7. The systolic array of claim 6, wherein: The one or more multiplexers provide output signals of a previous sub-processing element array to a next sub-processing element array in a first mode, and the sub-output buffer outputs the output signals of the previous sub-processing element array in a second mode.

8. A processing circuit comprising a systolic array, wherein: Systolic arrays include: a plurality of sub-processing element arrays, wherein each of the plurality of sub-processing element arrays is configured to receive an input signal and generate an output signal based on the received input signal; and a plurality of output direct path circuits configured to operate in a first mode and a second mode, wherein an output direct path circuit of the plurality of output direct path circuits receives an output signal of a previous sub-processing element array of the plurality of sub-processing element arrays, wherein the output direct path circuit provides an output signal to a next sub-processing element array of the plurality of sub-processing element arrays in the first mode, and wherein the output direct path circuit outputs the output signal in the second mode.

9. The processing circuit according to claim 8, wherein: Each of the plurality of output direct path circuits is disposed between two consecutive sub-processing element arrays of the plurality of sub-processing element arrays.

10. The processing circuit according to claim 8, wherein: A first time period for receiving a first input signal at a first sub-processing element array among the plurality of sub-processing element arrays overlaps with a second time period for receiving a second input signal at a second sub-processing element array among the plurality of sub-processing element arrays.

11. The processing circuit according to claim 8, wherein: The plurality of sub-processing element arrays are preloaded with weight values corresponding to the weight matrix in the first mode, and are preloaded with weight values corresponding to the compressed weight matrix in the second mode.

12. The processing circuit according to claim 8, wherein: The output direct path circuit outputs an output signal of a previous sub-processing element array to the restoration circuit in the second mode.

13. The processing circuit according to claim 8, wherein: The output direct path circuit includes a sub-output buffer and one or more multiplexers, wherein the sub-output buffer is configured to receive an output signal of a previous sub-processing element array as an input, and wherein the one or more multiplexers are each connected to the sub-output buffer.

14. The processing circuit according to claim 13, wherein: The one or more multiplexers provide output signals of a previous sub-processing element array to a next sub-processing element array in a first mode, and The sub-output buffer outputs an output signal of a previous sub-processing element array to the restoration circuit in the second mode.

15. An electronic device comprising: processing circuits, including systolic arrays, Among them, the systolic array includes: a plurality of sub-processing element arrays, wherein each of the plurality of sub-processing element arrays is configured to receive an input signal and generate an output signal based on the received input signal, and a plurality of output direct path circuits configured to operate in a first mode and a second mode, wherein an output direct path circuit of the plurality of output direct path circuits receives an output signal of a previous sub-processing element array of the plurality of sub-processing element arrays, wherein the output direct path circuit provides an output signal to a next sub-processing element array of the plurality of sub-processing element arrays in the first mode, and wherein the output direct path circuit outputs the output signal in the second mode.

16. The electronic device according to claim 15, wherein: Each of the plurality of output direct path circuits is disposed between two consecutive sub-processing element arrays of the plurality of sub-processing element arrays.

17. The electronic device according to claim 15, wherein: A first time period for receiving a first input signal at a first sub-processing element array among the plurality of sub-processing element arrays overlaps with a second time period for receiving a second input signal at a second sub-processing element array among the plurality of sub-processing element arrays.

18. The electronic device according to claim 15, wherein: The plurality of sub-processing element arrays are preloaded with weight values corresponding to the weight matrix in the first mode, and are preloaded with weight values corresponding to the compressed weight matrix in the second mode.

19. The electronic device according to claim 15, wherein: The output direct path circuit outputs an output signal of a previous sub-processing element array to the restoration circuit in the second mode.

20. The electronic device according to claim 15, wherein The output direct path circuit includes a sub-output buffer and one or more multiplexers, wherein the sub-output buffer is configured to receive an output signal of a previous sub-processing element array as an input, and wherein the one or more multiplexers are each connected to the sub-output buffer.

Images