Memory device and operation method thereof, and computing equipment

Abstract

A memory device and an operation method thereof, and a computing device are disclosed. The memory device comprises: a memory configured to store input data and filter data for a convolution operation; and a function processor configured to: in response to a read command of at least a portion of the input data and the filter data, transform the at least a portion of the data based on a parameter of the convolution operation during a clock cycle corresponding to the read command, and output a corresponding transformation result as transformed data.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2019-0115443 filed on September 19, 2019, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety for all purposes by reference. Technical Field

[0002] The following description relates to memory devices and methods. Background Art

[0003] Typically, a system for performing convolution operations or matrix operations may move data stored in a memory to a high-speed operation processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a neural processing unit (NPU)), and may store the resulting data in the memory. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0005] In one general aspect, a memory device includes: a memory configured to store input data and filter data for a convolution operation; and a functional processor configured to: in response to a read command of at least a portion of the input data and the filter data, transform the at least a portion of the data based on a parameter of the convolution operation during a clock cycle corresponding to the read command, and output a corresponding transformation result as transformed data.

[0006] The functional processor may include an input data transformation circuit, which is configured to: transform at least a portion of the input data and output the corresponding transformation result as the transformed input data, and the input data transformation circuit may be constructed based on the parameters of the convolution operation and the type of algorithm applied to transform the at least a portion of the input data to reduce the computational amount of the convolution operation.

[0007] The input transformation matrix may be determined based on both the parameters of the convolution operation and the type of the algorithm, and the input data transformation circuit may be configured to correspond to the input transformation matrix.

[0008] The functional processor may include a filter data transformation circuit, which is configured to: transform at least a portion of the filter data and output the corresponding transformation result as the transformed filter data, and the filter data transformation circuit may be constructed based on the parameters of the convolution operation and the type of algorithm applied to the at least a portion of the transformed filter data to reduce the computational amount of the convolution operation.

[0009] The filter transformation matrix may be determined based on both the parameters of the convolution operation and the type of the algorithm, and the filter data transformation circuit may be configured to correspond to the filter transformation matrix.

[0010] The filter data stored in the memory may include filter data transformed based on parameters of a convolution operation, and when the filter data is filter data transformed based on parameters of a convolution operation, the functional processor may be configured to output the transformed filter data without further transforming the transformed filter data.

[0011] The function processor may include a multiplier accumulator (MAC) configured to perform an operation between the transformed input data and the transformed filter data.

[0012] The functional processor may be configured to, in response to a write command of at least a portion of intermediate output data output by an operation between transformed input data and transformed filter data, transform the at least a portion of the intermediate output data based on parameters of the convolution operation during a clock cycle corresponding to the write command, and output the corresponding transformation result as the transformed intermediate output data.

[0013] The intermediate output data may include output data of an element-by-element multiplication between the transformed input data and the transformed filter data.

[0014] The functional processor may include an output data transformation circuit, which is configured to transform at least a portion of the intermediate output data and output the corresponding transformation result as the transformed intermediate output data, and the output data transformation circuit may be constructed based on the parameters of the convolution operation and the type of algorithm for transforming the intermediate output data to reduce the computational amount of the convolution operation.

[0015] The output transformation matrix may be determined based on the parameters of the convolution operation and the type of the algorithm, and the output data transformation circuit may be configured to correspond to the output transformation matrix.

[0016] Parameters of the convolution operation may be determined based on any one of the size of the output data, the size of the filter data, the size of the input data, the step interval, and the padding size, or any combination of any two or more thereof.

[0017] The functional processor may include either or both of an adder and a shifter.

[0018] The algorithm for transforming input data to reduce the computational complexity of the convolution operation may be a Winograd algorithm.

[0019] The algorithm for transforming input data to reduce the computational complexity of the convolution operation may be the Strassen algorithm.

[0020] The function processor may include a pruning circuit configured to output a predetermined value in response to a size of the transformed input data being less than or equal to a threshold value.

[0021] The function processor may be configured to calculate a value of a non-linear function for determining whether to activate the transformed filter data.

[0022] The function processor may include an operation processor configured to perform an operation using the transformed input data and the transformed filter data.

[0023] The operation processor may include a multiplier configured to perform element-by-element multiplication using the transformed input data and the transformed filter data.

[0024] The functional processor may include two or more of the following items: an input data transformation circuit configured to transform and output at least a portion of the input data; a filter data transformation circuit configured to transform and output at least a portion of the filter data; a multiplier configured to perform multiplication using the transformed input data and the transformed filter data; and an output data transformation circuit configured to transform and output at least a portion of the intermediate output data output by performing the multiplication.

[0025] In another general aspect, a computing device includes: a memory configured to store input data and filter data for a convolution operation; a functional processor configured to: in response to a read command of at least a portion of the input data and the filter data, transform the at least a portion of the data based on a parameter of the convolution operation during a clock cycle corresponding to the read command, and output a corresponding transformation result as transformed data; and a direct memory access (DMA) processor configured to: align and store the at least a portion of the data in the memory based on a connection relationship between the memory and the functional processor.

[0026] The functional processor may include an input data transformation circuit, which is configured to: transform at least a portion of the input data and output the corresponding transformation result as the transformed input data, and the input data transformation circuit may be constructed based on the parameters of the convolution operation and the type of algorithm applied to transform at least a portion of the input data to reduce the computational amount of the convolution operation.

[0027] The input transformation matrix may be determined based on a combination of parameters of the convolution operation and the type of the algorithm, and the input data transformation circuit may be configured to correspond to the input transformation matrix.

[0028] The functional processor includes a filter data transformation circuit, which is configured to: transform at least a portion of the filter data and output the corresponding transformation result as the transformed filter data, and the filter data transformation circuit can be constructed based on the parameters of the convolution operation and the type of algorithm applied to the at least a portion of the transformed filter data to reduce the computational amount of the convolution operation.

[0029] The computing device may further include: an operation processor configured to perform an operation using the transformed input data and the transformed filter data.

[0030] The operation processor may include any one of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a neural processing unit (NPU) and a field programmable gate array (FPGA), or any combination of any two or more thereof.

[0031] The functional processor may be configured to, in response to a write command of at least a portion of intermediate output data output by an operation using transformed input data and transformed filter data, transform the at least a portion of the intermediate output data based on parameters of the convolution operation during a clock cycle corresponding to the write command, and output the corresponding transformation result as the transformed intermediate output data.

[0032] The functional processor may include an output data transformation circuit, which is configured to transform at least a portion of the intermediate output data and output the corresponding transformation result as the transformed intermediate output data, and the output data transformation circuit may be constructed based on the parameters of the convolution operation and the type of algorithm for transforming the intermediate output data to reduce the computational amount of the convolution operation.

[0033] In another general aspect, an operation method of a processor-implemented memory device includes: receiving at least a portion of input data and filter data from a memory in response to a read command; and transforming the at least a portion of the data based on parameters of a convolution operation during a clock cycle corresponding to the read command, and outputting a corresponding transformation result as transformed data.

[0034] The output step may include: determining an input transformation matrix based on a combination of parameters of the convolution operation and the type of algorithm that transforms at least a portion of the input data to reduce the computational amount of the convolution operation; and transforming the at least a portion of the input data based on the input transformation matrix, and outputting the corresponding transformation result as the transformed input data.

[0035] The output step may include: determining a filter transformation matrix based on a combination of parameters of the convolution operation and at least a portion of the transform filter data to reduce the computational amount of the convolution operation; and transforming the at least a portion of the filter data based on the filter transformation matrix, and outputting the corresponding transformation result as transformed filter data.

[0036] The method may also include: in response to a write command of at least a portion of intermediate output data output by an operation between transformed input data and transformed filter data, transforming the at least a portion of the intermediate output data based on parameters of the convolution operation during a clock cycle corresponding to the write command, and storing the at least a portion of the transformed intermediate output data in a memory.

[0037] The steps stored in the memory may include: determining an output transformation matrix based on the parameters of the convolution operation and the type of algorithm for transforming at least a portion of the intermediate output data to reduce the computational complexity of the convolution operation; and transforming at least a portion of the intermediate output data based on the output transformation matrix, and storing at least a portion of the transformed intermediate output data in the memory.

[0038] In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the above method.

[0039] In another general aspect, a computing device includes: one or more processors configured to: in response to a read command of at least a portion of input data and filter data stored in a memory including a function in memory (FIM), transform the at least a portion of the data based on parameters of a convolution operation during a clock cycle corresponding to the read command, and output the corresponding transformation result as transformed data.

[0040] The memory may be configured to store input data and filter data.

[0041] The computing device may further include a direct memory access (DMA) processor configured to align and store the at least a portion of the data in the memory based on a connection relationship between the memory and the one or more processors.

[0042] The one or more processors may include a multiplier accumulator (MAC) configured to perform operations between transformed input data and transformed filter data.

[0043] Other features and aspects will be apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An example system for performing convolution operations or matrix operations is shown.

[0045] Figure 2 An example Winograd algorithm based on a one-dimensional (1D) convolution operation is shown.

[0046] Figure 3 An example Winograd algorithm based on a two-dimensional (2D) convolution operation is shown.

[0047] Figure 4 is a flow chart illustrating an example operation of a memory device.

[0048] Figure 5 An example memory device including pre-processed input data transformation circuitry for performing a ID convolution operation according to a Winograd algorithm is shown.

[0049] Figure 6 An example memory device including pre-processed filter data transformation circuitry for performing a ID convolution operation according to a Winograd algorithm is shown.

[0050] Figure 7 An example system for performing a 1D convolution operation according to the Winograd algorithm is shown.

[0051] Figure 8 An example memory device including output data transformation circuitry for performing post-processing of a ID convolution operation according to a Winograd algorithm is shown.

[0052] Fig. 9 An example method of aligning and storing data is shown.

[0053] Fig.10 An example method of transforming preprocessed filter data for a 2D convolution operation according to a Winograd algorithm is shown.

[0054] Fig.11 An example filter data transformation circuit is shown for performing preprocessing of a 2D convolution operation according to the Winograd algorithm.

[0055] Fig.12 An example transformation of preprocessed input data for a 2D convolution operation according to the Winograd algorithm is shown.

[0056] Fig.13 An example input data transformation circuit is shown for preprocessing to perform a 2D convolution operation according to the Winograd algorithm.

[0057] Fig.14An example system for performing a 2D convolution operation according to the Winograd algorithm is shown.

[0058] Fig.15 An example method of reducing information loss due to division is shown.

[0059] Fig.16 An example operation method of a memory device according to the Strassen algorithm is shown.

[0060] Fig.17 An example method of operation of a memory device including a multiplier accumulator (MAC) unit is shown.

[0061] Throughout the drawings and detailed description, like reference numerals denote like elements. The drawings may not be to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0062] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0063] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0064] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0065] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0066] Throughout the specification, when an element (such as a layer, region, or substrate) is described as being “on,” “connected to,” or “coupled to” another element, the element may be directly “on,” “connected to,” or “coupled to” another element, or one or more other elements may be present in between. Conversely, when an element is described as being “directly on,” “directly connected to,” or “directly coupled to” another element, there may be no other elements in between.

[0067] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0068] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0069] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0070] Examples may be configured as or in various types of products (e.g., personal computers (PCs), laptop computers, tablet computers, smart phones, televisions (TVs), smart electronic devices, smart vehicles, kiosks, and wearable devices). Examples are described below with reference to the accompanying drawings. Reference numerals in the drawings are always the same for the elements.

[0071] As will be clear after understanding the disclosure of the present application, the features of the examples described herein can be combined in various ways. In addition, although the examples described herein have various configurations, other configurations are feasible, as will be clear after understanding the disclosure of the present application. It is also noted that the use of the term "may" with respect to an example or embodiment (e.g., with respect to what an example or embodiment may include or implement) means that there is at least one example or embodiment that includes or implements such a feature, and all examples and embodiments are not limited thereto.

[0072] In an alternative system, a typical memory for storing data is separated from an operation processor for processing operations. In order to process the data stored in the memory, the data is moved to the operation processor and then processed in the operation processor. In order to process complex operations, the CPU uses many instructions and requires a relatively large amount of processing time. If a specific accelerator (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), and a digital signal processor (DSP)) is used as an operation processor, and in such an example there are operations that cannot be processed in the corresponding typical device, the data can be moved to the CPU and processed in the CPU. Therefore, the processing speed can be significantly reduced. In addition, in a typical system that requires pre-processing and post-processing (e.g., addition and subtraction), the operation processor may not be able to process the corresponding pre-processing or post-processing in a limited number of clock cycles or only one clock cycle.

[0073] Figure 1 An example system for performing convolution operations or matrix operations is shown.

[0074] Reference Figure 1 According to one or more embodiments, a system for performing a convolution operation or a matrix operation (also referred to as a "system according to an example" or simply referred to as a "system") may include a memory device 110 and a high-speed operation processor 140. The system according to a non-limiting example may be, for example, a computing device configured to perform a convolution operation or a matrix operation.

[0075] Convolution may include multiplying a single function with a value inverted from another function and integrating the multiplication result over an interval. In the example machine learning example, here, the convolution operation may represent an operation of selecting a filter corresponding to a given purpose and extracting a specific feature corresponding to the input data by scanning all regions of the input data using the selected filter. For example, the system may obtain output data by performing a convolution operation of the filter data relative to the input data, and each piece of data may be defined in a matrix form. When the data is defined in such a matrix form, the convolution operation may be performed as a matrix operation.

[0076] Matrix operations may include any possible arithmetic operations performed between multiple matrices. Non-limiting examples of such matrix operations include matrix addition and subtraction, scalar matrix multiplication, matrix multiplication, and element-by-element matrix multiplication. In addition, matrix operations may include operations that can be expressed in matrix form, such as linear equations.

[0077] Convolution operations can be characterized as a combination of matrix additions and subtractions and matrix multiplications. In such examples, the amount of time and power used for matrix multiplications can be significantly greater than the amount of time and power used for matrix additions and subtractions. From a system perspective, reducing the number of matrix multiplication operations can be a way to increase convolution operation processing speed and reduce the power consumption incurred when performing such convolution operations.

[0078] In addition, a typical system for performing a convolution operation moves raw data stored in a memory to a high-speed operation processor (such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a neural processing unit (NPU), a field programmable gate array (FPGA)), and such a high-speed operation processor processes the data and then stores the processed data in the memory. In a typical system, the memory storing the data and the operation processor processing the operation may be separated from each other. In order to process the data stored in the memory, the data may be moved to the operation processor and processed at the operation processor. Therefore, a relatively large amount of time is used to process the data.

[0079] According to one or more embodiments, a system according to an example may use an algorithm that transforms data and performs a convolution operation to reduce the amount of computation for the convolution operation. For example, such a system may reduce the multiplication complexity of the convolution operation, for example, by using a Winograd algorithm to reduce the number of multiplications required for the convolution operation. Alternatively, such a system may use a Strassen algorithm to reduce the multiplication complexity of the convolution operation. In examples using such algorithms (such as a Winograd algorithm or a Strassen algorithm), preprocessing or postprocessing (such as data conversion) may be used. Conversely, the system may use an operation processor to perform corresponding preprocessing or postprocessing.

[0080] According to one or more embodiments, the system according to the example may use a memory device 110 embedded with a functional module 130, and the memory device 110 may use the functional module 130 to perform pre-processing or post-processing of an algorithm. As a non-limiting example, the functional module 130 may correspond to a functional processor. In this way, the processing speed of a convolution operation or a matrix operation may be increased, and at the same time, the power efficiency of the high-speed operation processor 140 may be increased. In more detail, before the high-speed operation processor 140 processes data, the functional module 130, for example, included in the memory device 110 in the form of a circuit, may perform pre-processing or post-processing of the algorithm. As a non-limiting example discussed above, the Winograd algorithm or the Strassen algorithm may be used for processing. Therefore, by using the functional module 130 as discussed in further detail above, since some or all of the convolution operations and / or matrix operations may be performed by the memory device 110, or some or all of the convolution operations and / or matrix operations may be separated from one or more of the aforementioned example high-speed operation processors, a larger amount of data may be processed during the same amount of time.

[0081] The memory device 110 may include a function in memory (FIM) capable of performing data storage and data processing. Such a FIM may include hardware implementation. The functional module 130 may perform pre-processing or post-processing of an algorithm in a clock cycle corresponding to a read command or a write command. Because the functional module 130 can perform pre-processing or post-processing of an algorithm in a clock cycle corresponding to a read command or a write command, the system according to the example can reduce the amount of time for pre-processing or post-processing of the aforementioned typical system method.

[0082] In one example, a clock cycle corresponding to a read command or a write command may correspond to a single clock cycle for operation of the memory device 110. In this example, circuits other than the multiplier may be included in the functional module 130 to complete the processing of the functional module 130 in a single cycle, which is described in further detail below.

[0083] In another example, the clock cycle corresponding to the read command or the write command may correspond to multiple clock cycles for operation of the memory device 110. In this example, the functional module 130 may include a multiplier. A pipeline scheme may be applied to the memory device 110 to enhance successful management of latency, which is described in further detail below.

[0084] The memory device 110 may include a memory unit 120 and a functional module 130. Here, the memory unit 120 may also be referred to as a memory module. The memory unit 120 may store input data, filter data, and output data. The input data may represent data on which a convolution operation is performed, and may be, for example, pixel data of an image represented in the form of a two-dimensional (2D) function. The filter data may represent data related to a filter for detecting features of the input data. For example, the filter data may be a common parameter for detecting features of an image. Therefore, as a non-limiting example of related terms representing filters, the filter may be alternatively referred to as a kernel, a mask, a window, etc. In general, the filter may be defined in a square matrix (such as a matrix having a size of (4, 4) or a size of (3, 3), but these are only non-limiting examples, and square matrices of other sizes may be used in other examples, and there may be multiple filters. The system may generate a feature map by performing a convolution operation on the input data using a filter at a specified interval. In such an example, the specified interval for moving or sliding the filter on the input data (also referred to as making the filter tour) may also be referred to as a step size.

[0085] As a non-limiting example, the functional module 130 may include a transformation circuit configured to process a transformation equation used by the Strassen algorithm, the Winograd algorithm, or a modified algorithm thereof. The transformation circuit may include an input data transformation circuit 131, a filter data transformation circuit 133, and an output data transformation circuit 135. In response to a read command of at least a portion of the input data, the input data transformation circuit 131 may transform at least a portion of the input data in a clock cycle corresponding to the read command and output it for preprocessing of the algorithm. The data transformed by the input data transformation circuit 131 may also be referred to as transformed input data.

[0086] In response to the execution of the read command of at least a portion of the filter data, the filter data conversion circuit 133 can convert at least a portion of the filter data in a clock cycle corresponding to the read command and output it for preprocessing of the algorithm. The data converted by the filter data conversion circuit 133 can also be referred to as converted filter data.

[0087] The high-speed operation processor 140 may receive the transformed input data and the transformed filter data, and may perform the remaining operations according to the algorithm. As a result, the high-speed operation processor 140 may generate intermediate output data. The high-speed operation processor 140 may include at least one of a CPU, a GPU, a DSP, an NPU, and an FPGA as discussed in further detail above, but these are merely non-limiting examples. The high-speed operation processor 140 may also be embedded in a memory having at least one of an input data conversion circuit and a filter data conversion circuit. For example, the high-speed operation processor 140 may be embedded in the memory device 110 or in the functional module 130 of the memory device 110.

[0088] In response to a write command of at least one of the intermediate output data, the output data transformation circuit 135 may generate output data for post-processing of the algorithm by transforming at least a portion of the intermediate output data in a clock cycle corresponding to the write command, and may store the generated output data in the memory unit 120. The output data may be final result data of a convolution operation performed on the input data and the filter data.

[0089] In the following, reference is made to Figure 2 and Figure 3 The Winograd algorithm is described in more detail in the examples of Figure 4 An example of describing the operation method of the memory device is shown in FIG. Figures 5 to 8 The example of describing the operation method of the memory device according to the Winograd algorithm in a one-dimensional (1D) convolution operation, referring to Fig. 9 An example description of a computing device including a direct memory access (DMA) module configured to align and store data, with reference to Figures 10 to 14 The example of describing the operation method of the memory device according to the Winograd algorithm in a two-dimensional (2D) convolution operation, referring to Fig.15 An example of reducing the information loss caused by division is described in Fig.16 The example of describes a method of operation of a memory device according to the Strassen algorithm.

[0090] Figure 2 An example of the Winograd algorithm based on a 1D convolution operation is shown.

[0091] There are several ways to perform a convolution operation. If you apply an inverse Fourier transform (IFT) function to the result of multiplying the two Fourier transform functions on which the convolution operation is performed, you can get the same result as the result of convolving the original functions, as shown in Equation 1 below:

[0092] Equation 1

[0093]

[0094] Referring to Equation 1, if the operation is performed in the frequency domain instead of performing convolution in the time domain, the number of multiplications may be reduced, which may result in an increase in the processing speed of implementing the overall operation and an increase in the power efficiency of the overall operation.

[0095] Reference Figure 2 For example, a typical 1D convolution operation method can perform four input data k 0 , k 1 , k 2 and k 3 And 3 filter data w 0 、w 1 and w 2 Perform convolution operation and output 2 output data r 0 and r 1 , as shown in Equation 2 below:

[0096] Equation 2

[0097] r 0 =(k 0 ·w 0 )+(k 1 ·w 1 )+(k 2 ·w 2 )

[0098] r 1 =(k 1 ·w 0 )+(k 2 ·w 1 )+(k 3 ·w 2 )

[0099] Referring to Equation 2, an alternative 1D convolution operation method may use 6 multiplications.

[0100] According to the Winograd algorithm shown in Equation 3, m 0 To m 3 The combination of is used to modify and represent the output data, where Equation 3 is presented below:

[0101] Equation 3

[0102]

[0103] F(m,n) is a symbol indicating a process with m pieces of output data and n pieces of filter data using a finite impulse response (FIR) filter based on a convolution operation. Here, m 0 To m 3 It can be expressed by the following equation 4:

[0104] Equation 4

[0105]

[0106]

[0107] The 1D convolution operation according to the Winograd algorithm can use 4 multiplications. That is, the number of multiplications is further reduced compared to the 6 multiplications used in the alternative method.

[0108] Generally, the method of performing 1D convolution operation according to the Winograd algorithm may include four stages: 0 , k 1 , k 2 and k 3 Transform to k 0 -k 2 , k 1 +k 2 , k 1 -k 3 and k 2 -k 1 The first stage; filter data w 0 、w 1 and w 2 Transformed into (w 0 ,w 0 +w 1 +w 2 ) / 2 and (w 2 ,w 0 -w 1 +w 2 ) / 2; m is calculated by performing element-by-element multiplication on the transformed input data and the transformed filter data 0 To m 3 The third stage; and the use of m 0 To m 3 Output output data 0 and r 1 The four stages can be arranged as shown in Equation 5 below:

[0109] Equation 5

[0110] S=A T [[(Gw T )⊙(C T k T )]

[0111] In equation 5, S = (r 0 ,r 1 ), w=(w 0 ,w 1 ,w 2), and k=(k 0 ,k 1 ,k 2 ,k 3 ).

[0112] The input data k 0 , k 1 , k 2 and k 3 Transform to k 0 -k 2 , k 1 +k 2 , k 1 -k 3 and k 2 -k 1 The first stage may correspond to executing (C T k T ). In such an example, C represents an input transformation matrix for transforming input data and can be represented by the following equation 6:

[0113] Equation 6

[0114]

[0115] The filter data w 0 、w 1 and w 2 Transformed into (w 0 ,w 0 +w 1 +w 2 ) / 2 and (w 2 ,w 0 -w 1 +w 2 The second stage of ) / 2 may correspond to executing (Gw T ). In such an example, G represents a filter transformation matrix for transforming filter data and can be represented by the following Equation 7:

[0116] Equation 7

[0117]

[0118] m is calculated by performing an element-wise multiplication of the transformed input data and the transformed filter data. 0 To m 3 The third stage may correspond to performing the (Gw T )⊙(CTk T ) is calculated to output the intermediate output data, and may correspond to performing multiplication in the frequency domain according to Equation 1. Performing element-by-element multiplication may mean performing an operation of multiplying elements existing at the same position in two matrices.

[0119] Use m 0 To m 3 To output the output data r 0 and r 1 The fourth stage may correspond to executing A of Equation 5 T with(Gw T )⊙(C T k T ), and may correspond to transforming the frequency domain result in equation 1 to the time domain. In such an example, A represents the n output transformation matrices for transforming the intermediate output data, and may be represented by the following equation 8:

[0120] Equation 8

[0121]

[0122] The input data k 0 , k 1 , k 2 and k 3 Transform to k 0 -k 2 , k 1 +k 2 , k 1 -k 3 and k 2 -k 1 The first stage and filter data w 0 、w 1 and w 2 Transformed into (w 0 ,w 0 +w 1 +w 2 ) / 2 and (w 2 ,w 0 -w 1 +w 2 The second stage of ) / 2 can correspond to the above preprocessing process, using m 0 To m 3 To output the output data r 0 and r 1 The fourth stage may correspond to the above-described post-processing process. A system according to such an example may perform the first and second stages in the memory device 110, may perform the third stage in the high-speed operation processor 140, and may perform the fourth stage in the memory device 110 again.

[0123] Figure 3 An example of a Winograd algorithm based on a two-dimensional (2D) convolution operation is shown.

[0124] Reference Figure 3As shown in the example indicators 310 and 320 of , a typical 2D convolution operation can be performed by applying a 3×3 matrix constituting a filter to a 4×4 matrix of input data. In this example, 36 multiplications can be used.

[0125] The 2D convolution operation according to the Winograd algorithm may be performed by iterating the 1D convolution operation according to the Winograd algorithm twice, which may be represented by the following Equation 9:

[0126] Equation 9

[0127] S=A T [((GwG T )⊙M)⊙(C T kC)]A

[0128] For example, referring to indicator 330, in the case of applying a 3×3 filter to 4×4 input data, a 2D convolution operation according to the Winograd algorithm may be represented by Equation 10 and may use only 16 multiplications, where Equation 10 is provided below:

[0129] Equation 10

[0130] S=A T [(GwG T )⊙(C T kC)]A

[0131] In such an example, the input transformation matrix, the filter transformation matrix, and the output transformation matrix may be represented by Equations 11, 12, and 13, respectively, provided below:

[0132] Equation 11

[0133]

[0134] Equation 12

[0135]

[0136] Equation 13

[0137]

[0138] When processing input data having a size of m×n using a filter having a size of r×s, a typical 2D convolution operation may use m×n×r×s multiplications, and a 2D convolution operation according to the Winograd algorithm may use (m+r-1)×(n+s-1) multiplications.

[0139] In the example of performing convolution operations according to the Winograd algorithm, the number of matrix additions can be increased and the number of matrix multiplications can be correspondingly reduced. By inducing this tradeoff, the overall operation processing speed can be increased and power consumption can be reduced, because the impact of the increase in matrix additions can be overwhelmed by the corresponding reduction in matrix multiplications.

[0140] Figure 4 is a flowchart illustrating an example of an operation method of a memory device.

[0141] Reference Figure 4 For example, operation 410 and operation 420 may be performed by Figure 1 As a non-limiting example, Figure 4 The example operation method can be applied to any system including memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), parametric random access memory (PRAM), magnetic random access memory (MRAM), and flash memory), regardless of the type of memory.

[0142] In operation 410 , the function module 130 may receive at least a portion of data among input data and filter data from the memory unit 120 in response to a read command.

[0143] In operation 420, the functional module 130 may transform and output at least a portion of the data based on the parameters of the convolution operation in a clock cycle corresponding to the read command. The functional module 130 may transform and output at least a portion of the input data based on the input transformation matrix. For example, the functional module 130 may include an input data transformation circuit 131 designed to correspond to the input transformation matrix, and the input data transformation circuit 131 may transform at least a portion of the input data and may output the transformed input data.

[0144] The input transformation matrix may be determined based on a combination of parameters of the convolution operation and the type of algorithm for transforming at least a portion of the input data. As a non-limiting example, the parameters of the convolution operation may be determined based on any one of the size of the output data, the size of the filter data, the size of the input data, the step interval, and the padding size, or any combination of any two or more. The algorithm may represent an algorithm that transforms data to reduce the multiplication complexity of the convolution operation. As a non-limiting example, the aforementioned Winograd algorithm, Strassen algorithm, or a modified algorithm thereof may be applied. For example, in response to performing a convolution operation on F(2,3) using the Winograd algorithm, the input transformation matrix may be represented by Equation 6. In addition, in response to performing a convolution operation on F(2×2,3×3), the input transformation matrix may be represented by Equation 11.

[0145] The functional module 130 may transform and output at least a portion of the filter data based on the filter transformation matrix. For example, the functional module 130 may include a filter data transformation circuit 133 designed to correspond to the filter transformation matrix. The filter data transformation circuit 133 may transform at least a portion of the filter data and may output the transformed filter data. The filter transformation matrix may be determined based on a combination of parameters of the convolution operation and the type of algorithm for transforming at least a portion of the filter data. For example, in response to performing a convolution operation on F(2,3) using the Winograd algorithm, the filter transformation matrix may be represented by Equation 7. In addition, in response to performing a convolution operation on F(2×2,3×3), the filter transformation matrix may be represented by Equation 12.

[0146] Using the multiplexer, the memory device 110 may or may not employ the filter data transformation circuit 133. For example, since the filter data does not change frequently, the transformed filter data pre-calculated based on the filter transformation matrix may be stored in the memory unit 120, and the memory device 110 may directly load and use the stored filter data using the multiplexer.

[0147] In response to a write command of at least a portion of the intermediate output data output by the operation between the transformed input data and the transformed filter data, the functional module 130 may transform at least a portion of the intermediate output data based on the parameters of the convolution operation in a clock cycle corresponding to the write command, and may store the transformed intermediate output data in the memory unit 120.

[0148] For example, the functional module 130 may include an output data transformation circuit 135 designed to correspond to the output transformation matrix. The output data transformation circuit 135 may transform at least a portion of the intermediate output data and may output the output data. The output transformation matrix may be determined based on a combination of parameters of the convolution operation and the type of algorithm that transforms at least a portion of the intermediate output data. For example, in response to performing a convolution operation on F(2,3) using the Winograd algorithm, the output transformation matrix may be represented by Equation 8. In addition, in response to performing a convolution operation on F(2×2,3×3), the output transformation matrix may be represented by Equation 13.

[0149] Functional module 130 can perform pre-processing or post-processing of the algorithm in a clock cycle corresponding to a read command or a write command. Pre-processing of the algorithm may include generating transformed input data and / or transformed filter data, and post-processing of the algorithm may include transforming intermediate output data into output data. Because functional module 130 can perform pre-processing or post-processing of the algorithm in a clock cycle corresponding to a read command or a write command, the system according to the example can reduce the amount of time used for pre-processing and post-processing of the algorithm per cycle compared to a typical system.

[0150] If a single bus is shared by input data and output data, the memory device 110 may determine whether to use the input data conversion circuit 131 or the output data conversion circuit 135 based on a signal identifying an input command or an output command. For example, if the bus is used as an input port based on the signal, the memory device 110 may use the input data conversion circuit 131. Conversely, if the bus is used as an output port based on the signal, the memory device 110 may use the output data conversion circuit 135 instead.

[0151] Reference above Figures 1 to 4 The operation method of the memory device 110 capable of improving the operation speed of the convolution operation is further described in detail. Figures 5 to 8 The example further describes in detail the overall configuration of the functional modules for improving the operation speed of the 1D convolution operation, and refers to Figures 10 to 15 The example further describes in detail the overall configuration of the functional modules used to improve the computing speed of the 2D convolution operation.

[0152] Figure 5 An example of a memory device including a pre-processed input data transformation circuit for performing a ID convolution operation according to a Winograd algorithm is shown.

[0153] Reference Figure 5 In the example of FIG. 5 , the memory device may include a memory unit 510 and a functional module. Figures 1 to 4 The example described is applicable to Figure 5 , and for the sake of brevity, repeated descriptions are omitted.

[0154] The system according to an example may perform a convolution operation on F(2,3) using a Winograd algorithm. The memory unit 510 may store input data based on a 4-data unit which is a unit containing 4 pieces of data.

[0155] The functional module may include an input data transformation circuit 520. The input data transformation circuit 520 may transform the input data k0, k1, k2, and k3 into k0-k2, k1+k2, k1-k3, and k2-k1. The input data transformation circuit 520 may be designed to correspond to the input transformation matrix of equation 6. For example, the input data transformation circuit 520 may be configured as a combination of multiple adders corresponding to the input transformation matrix. In the case of using 2's complement or 1's complement representation to represent negative numbers, the adder may be used as an adder-subtractor element.

[0156] Since the input data transformation circuit 520 may be configured using only a combination of a plurality of adders having a relatively simple configuration and a relatively fast speed, the input data transformation circuit 520 may have sufficient speed to transform and output input data in a clock cycle corresponding to a read command.

[0157] However, Figure 5 The input data transformation circuit 520 of the example is provided only as a non-limiting example. Various circuits corresponding to the input transformation matrix can be used as the input data transformation circuit 520.

[0158] Figure 6 An example of a memory device including a pre-processed filter data transformation circuit for performing a ID convolution operation according to a Winograd algorithm is shown.

[0159] Reference Figure 6 In a non-limiting example, the memory device may include a memory unit 610 and a functional module.

[0160] The system according to the example may perform a convolution operation on F(2,3) using a Winograd algorithm. The memory unit 610 stores filter data based on 3-data units which are units of 3 pieces of data.

[0161] The functional module may include a filter data conversion circuit 620. The filter data conversion circuit 620 may convert the filter data w0, w1, and w2 into (w0, w0+w1+w2) / 2 and (w2, w0-w1+w2) / 2. The filter data conversion circuit 620 may be designed to correspond to the filter conversion matrix of equation 7. For example, the filter data conversion circuit 620 may be configured as a combination of a shifter and an adder corresponding to the filter conversion matrix. Constant division may be performed using a shifter. In an example, the filter data may be referred to as a weight, and the filter data conversion circuit 620 may be referred to as a weight conversion circuit.

[0162] Since the filter data conversion circuit 620 can be configured using only a combination of shifters and adders, thereby having a relatively simple configuration and a relatively fast speed, the filter data conversion circuit 620 can have sufficient speed to convert and output the filter data during a clock cycle corresponding to a read command.

[0163] here, Figure 6 The example filter transformation circuit 620 of FIG. 6 is provided only as a non-limiting example. According to various non-limiting examples, various circuits corresponding to the filter transformation matrix may operate under the role of the filter transformation circuit 620 .

[0164] Figure 7 An example of a system for performing a 1D convolution operation according to the Winograd algorithm is shown.

[0165] Reference Figure 7 In an example of, the system may include a high-speed operation processor 710, and the high-speed operation processor 710 may calculate the intermediate output data m0 to m3 by performing element-by-element multiplication on the input data transformed by the input data transformation circuit 520 and the filter data transformed by the filter data transformation circuit 620 as discussed in further detail above. Here, m0 to m3 may be represented by the above equation 4.

[0166] In a non-limiting example, the high-speed operation processor 710 may be embedded in a memory device. For example, the high-speed operation processor 710 may be a multiplier. The multiplier may be embedded in a memory device and may also perform element-by-element multiplication on the transformed input data and the transformed filter data. Although as a non-limiting example, Figure 7 The memory unit 510 and the memory unit 610 are shown to be separate from each other, but as another non-limiting example, a single memory unit can also be used to read the input data and the filter data at the same time. For example, when configuring the memory interface, the data can usually be read based on the size unit of the data bus (e.g., the number of lines of the data bus). In such an example, since the number of bits of each input data or each filter data can be less than the number of lines of the data bus, when reading the data at one time, multiple pieces of data can be read at the same time.

[0167] Optionally, the number of channels of the data bus may be increased. For example, in a non-limiting example of using two 4GB DRAMs as 2 channels, the size of the data bus may be doubled accordingly, compared to using one 8GB DRAM as 1 channel. In this non-limiting example, a DMA module may be used to appropriately store input data and filter data in two channels. Fig. 9 The example further describes the method of aligning and storing data.

[0168] Figure 8 An example of a memory device including an output data transformation circuit for performing post-processing of a 1D convolution operation according to a Winograd algorithm is shown.

[0169] Reference Figure 8 In the example of , the memory device may include a memory unit 810 and a functional module.

[0170] The system according to an example may perform a convolution operation on F(2,3) using a Winograd algorithm. The memory unit 810 may store output data based on a 2-data unit which is a unit of 2 pieces of data.

[0171] The functional module may include an output data conversion circuit 820. The output data conversion circuit 820 may convert the intermediate output data m0, m1, m2, and m3 into r0 and r1. The output data r0 and r1 may be stored in the memory unit 810 after conversion.

[0172] The output data transformation circuit 820 may be designed to correspond to the output transformation matrix of Equation 8. For example, as a non-limiting example, the output data transformation circuit 820 may be configured as a combination of a plurality of adders corresponding to the output transformation matrix.

[0173] Because in a non-limiting example, the output data transformation circuit 820 can be configured to use only a combination of multiple adders which can result in a relatively simple configuration and a faster speed, the output data transformation circuit 820 can have sufficient speed to transform and output the intermediate output data during the clock cycle corresponding to the write command.

[0174] However, Figure 8 The output transformation circuit 820 of the example is provided only as a non-limiting example. Various circuits corresponding to the output transformation matrix can be used as the output transformation circuit 820.

[0175] Although not shown, a memory device according to various non-limiting examples may include any one of an input data transformation circuit, a weight transformation circuit, a multiplier, and an output data transformation circuit, or any combination of any two or more. For example, a memory device may include one of an input data transformation circuit, a weight transformation circuit, a multiplier, and an output data transformation circuit, or may include two or more of an input data transformation circuit, a weight transformation circuit, a multiplier, and an output data transformation circuit. According to one example, a memory device may also include all input data transformation circuits, weight transformation circuits, multipliers, and output data transformation circuits.

[0176] In response to a request for a high-frequency operation, the operation speed of the memory device can be increased by a pipeline scheme. For example, the memory device can use such a pipeline scheme to process memory reads, input data transformations, weight transformations, multiplications, output data transformations, and memory writes. If an operation with a low frequency is available, the operation of the memory device can be processed during a single clock cycle without using a pipeline.

[0177] The memory device may further include a port configured to stop a read / write signal to block requests from outside the memory device. For example, the memory device may block requests from outside by outputting a busy signal via a corresponding port.

[0178] The memory device may also include a counter configured to prevent reading / writing during a specific time period of the memory device. For example, a busy signal may be output during a time period set by the counter. Therefore, there may also be a corresponding time period set by the counter during which a busy signal is not output.

[0179] As a non-limiting example, the memory device may read the corresponding values ​​from the area where the input and weight are stored, and may perform input / output data transformation operations, multiplication operations, and / or output storage operations simultaneously or at time intervals. For example, the above-mentioned various operations may be implemented by performing read and write processing with different addresses using a dual-port SRAM.

[0180] Fig. 9 An example showing a method of aligning and storing data.

[0181] Reference Fig. 9 According to the example, the system according to the example may also include a DMA module or device. As a non-limiting example, the DMA device may represent hardware and / or a combination of hardware and software that enables a peripheral device (e.g., a hard disk, a graphics card, a network card, and a sound card) to read or write memory by direct access to the memory, and may be a function supported by an internal bus of the computer.

[0182] In one example, the DMA device can be used to align input data and filter data to perform the Winograd algorithm and output the data accordingly. Before using the data to perform the Winograd algorithm process, each piece of data needs to be aligned in a corresponding order by the DMA device.

[0183] In another example, the data movement function may be dynamically configured and used in an FPGA, and may also be generated and used using an application specific integrated circuit (ASIC).The data movement function may be executed by a CPU.

[0184] The DMA device or FPGA may include a buffer inside. By using such a buffer, the DMA device or FPGA can read a plurality of pieces of data and then record the data in a desired memory location in a desired order.

[0185] Fig.10 An example of a method of transforming preprocessed filter data for a 2D convolution operation according to a Winograd algorithm is shown.

[0186] The memory device according to a non-limiting example may transform and output at least a portion of the filter data in real time based on the filter transformation matrix, or according to another non-limiting example, may store pre-calculated transformed filter data in the memory unit.

[0187] If the filter data is reusable, the filter data may be transformed and stored in response to storing the initial filter data value. In this example, because no additional calculations are required for secondary use, the amount of power used for calculations may be saved, and the amount of time used to read commands may be reduced.

[0188] The system according to the example may perform a convolution operation on F(2×2, 3×3) using a Winograd algorithm. The memory device may transform the filter data w0 to w8 through the functional module and may store the transformed filter data in the memory unit.

[0189] In more detail, the filter data and the filter transformation matrix can be expressed by the following equations 14 and 15, respectively:

[0190] Equation 14

[0191]

[0192] Equation 15

[0193]

[0194] The transformed filter data can be obtained by executing (GwG) of Equation 10 T ) to obtain, and can be expressed by the following Table 1:

[0195] Table 1

[0196]

[0197] Reference Fig.10 In the example of , the memory device may transform the filter data w0 to w8 by using the functional module, and may pre-store the transformed filter data of Table 1 in the memory unit. Alternatively, the memory device may transform and output at least a portion of the filter data in real time based on using the filter transformation matrix.

[0198] Fig.11 An example of a preprocessed filter data transformation circuit for performing a 2D convolution operation according to the Winograd algorithm is shown.

[0199] Reference Fig.11 For example, the filter data transformation circuit may perform preprocessing on the filter data transformation of the convolution operation performed on F(2×2,3×3) using the Winograd algorithm. The filter data transformation circuit may be designed to correspond to Fig.10 For example, the filter data transformation circuit can be configured as a combination of a shifter and an adder corresponding to the operation associated with the filter transformation matrix. Therefore, a shifter can be used to perform constant division.

[0200] Because the filter data transformation circuit can be configured to use only a combination of shifters and adders, which can result in the filter data transformation circuit having a relatively simple configuration and a faster speed, the filter data transformation circuit can have sufficient speed to transform and output the filter data during the clock cycle corresponding to the read command.

[0201] here, Fig.11 The filter data conversion circuit of the example is provided only as a non-limiting example. In other non-limiting examples, various circuits corresponding to the filter conversion matrix can be used as the filter data conversion circuit.

[0202] Fig.12 An example of preprocessed transformed input data for a 2D convolution operation according to the Winograd algorithm is shown.

[0203] The system according to the example may perform a convolution operation on F(2×2, 3×3) using a Winograd algorithm. The memory device may transform and output input data k0 to k15 by using a functional module.

[0204] In more detail, the input data and the input transformation matrix may be expressed by the following Equations 16 and 17, respectively.

[0205] Equation 16

[0206]

[0207] Equation 17

[0208]

[0209] The transformed input data can be obtained by executing Equation 10 (C T kC) to obtain, and can be expressed by the following Table 2:

[0210] Table 2

[0211]

[0212] Reference Fig.12 In the example of , the memory device may transform and output input data k0 to k15 by using the function module.

[0213] Fig.13 An example of a preprocessed input data transformation circuit for performing a 2D convolution operation according to the Winograd algorithm is shown.

[0214] Reference Fig.13 For example, the input data transformation circuit may perform preprocessing on the input data transformation of the convolution operation to be performed on F(2×2,3×3) using the Winograd algorithm. The input data transformation circuit may be designed to correspond to Fig.12 For example, as a non-limiting example, the input data transformation circuit can be configured as a combination of multiple adders corresponding to the input transformation matrix.

[0215] Because the input data transformation circuit can be configured to use only a combination of multiple adders, thereby providing an input data transformation circuit with a relatively simple configuration and a faster speed, the input data transformation circuit can have sufficient speed to transform and output the input data during the clock cycle corresponding to the read command.

[0216] However, Fig.13 The input data transformation circuit of the example is provided only as a non-limiting example. In other non-limiting examples, various circuits corresponding to the input transformation matrix can be used as the input data transformation circuit.

[0217] Fig.14 An example of a system for performing a 2D convolution operation according to the Winograd algorithm is shown.

[0218] In one example, a high-speed computing processor can Fig.13 An example of an input data conversion circuit converts the input data and Fig.11 The filter data conversion circuit of the example performs element-by-element multiplication on the filter data converted to calculate the intermediate output data m0 to m15.

[0219] Reference Fig.14 In the example of, the functional module may include an output data conversion circuit. The output data conversion circuit may convert the intermediate output data m0 to m15 into output data r0 to r4. Once the output data r0 to r4 are generated, the output data r0 to r4 may be stored in the memory unit. In detail, the output data may be represented by the following Table 3:

[0220] Table 3

[0221]

[0222] The output data transformation circuit can be designed to correspond to the output transformation matrix. For example, the output data transformation circuit can be configured using a combination of multiple adders corresponding to the operations associated with the output transformation matrix, although this is a non-limiting example, other configurations can be used for the output data transformation circuit in other examples.

[0223] Since the output data conversion circuit can be configured to use only a combination of multiple adders with a relatively simple configuration and a relatively fast speed, the output data conversion circuit can have a sufficient operating speed to be able to convert and output intermediate output data during a clock cycle corresponding to a write command.

[0224] here, Fig.14 The output data transformation circuit of the example is provided only as a non-limiting example. In other non-limiting examples, various circuits corresponding to the output transformation matrix can be used as the output data transformation circuit.

[0225] Fig.15 An example of a method of reducing information loss due to division is shown.

[0226] In one example, because the filter transformation matrix may be in the form of rational numbers rather than integers, the filter data transformation circuit may need to be able to perform constant division using a shifter.

[0227] During the process of the filter data transformation circuit using the shifter to perform constant division, information loss may occur. In one example, the information loss due to the division may be reduced by adjusting the relationship between the filter transformation matrix and the output transformation matrix.

[0228] If the operation is performed without information loss and then the division is performed, the information loss due to the division can be further reduced compared to performing the convolution operation using data that loses information due to the division. In more detail, all elements of the filter transformation matrix are made integers, the intermediate output data can be calculated using the filter transformation matrix multiplied by a constant k, and the intermediate output data can be transformed using the filter transformation matrix divided by such a constant k.

[0229] Reference Fig.15 For example, a filter transformation matrix in which all elements are integers can be generated by multiplying the filter transformation matrix by 2, and the filter transformation matrix can be used to calculate the intermediate output data. The intermediate output data can be transformed using the filter transformation matrix divided by 2. In this way, information loss due to division can be reduced.

[0230] Fig.16 An example of an operation method of a memory device according to the Strassen algorithm is shown.

[0231] In one example, the Strassen algorithm represents an algorithm capable of reducing the complexity of matrix multiplication. If two matrices each having a size of n×n are multiplied, then approximately O(n×n) can be used according to an alternative method. 3 ) time. According to the Strassen algorithm, it can be done in about O(n 2.807 ) time.

[0232] Reference Fig.16 For example, if the Strassen algorithm is applied, 8 multiplications can be reduced to 7 multiplications. In the intermediate operation results required to calculate the intermediate output data m1 to m7, the operation associated with the A data (e.g., input data) can be processed in the memory in which the A data is stored. Then, the operation associated with the intermediate operation results of the A data and the B data (e.g., filter data) can be performed element-by-element multiplication. After storing the A data and the B data in a single memory, Fig.16 All operations shown in the examples or a portion of the above operations may be processed in the memory. In various non-limiting examples, the operation result C as the output data 1,1 To C 2,2 Additional storage space internal or external to the memory may be used.

[0233] Fig.17 An example of an operation method of a memory device including a multiplier accumulator (MAC) unit or only a MAC is shown.

[0234] Reference Fig.17 According to the example, the memory device according to the example may include a MAC, and may also include any one of an input data transformation circuit, a filter data transformation circuit, and an output data transformation circuit configured by selectively using a multiplier and an adder included in the MAC, or any combination of any two or more thereof.

[0235] Alternatively, any one of the input data transform circuit, the filter data transform circuit, and the output data transform circuit, or any combination of any two or more thereof may be configured by adding additional multipliers and adders to the multipliers and adders included in the MAC.

[0236] If the conversion circuit is configured using the MAC as above, a memory device capable of processing even the Winograd algorithm in addition to matrix-by-matrix multiplication, matrix-vector multiplication, and vector-by-vector multiplication can be configured.

[0237] Although not shown, if the size of the transformed input data is less than or equal to the threshold value, the function module may include a pruning circuit configured to output a predefined value.

[0238] For example, if when the input value of the algorithm is read from the memory unit through such a pruning circuit, the transformed value is less than or equal to the reference value, the functional module may output a 0 value to the output port of the memory. Therefore, the memory device can improve the processing speed by embedding the pruning circuit into the functional module. The memory device may include a register configured to store such a pruning reference value in the memory.

[0239] The functional module may further include a nonlinear function circuit configured to determine whether to activate the transformed filter data. By embedding such a nonlinear circuit into the functional module, the memory device may increase the processing speed without additional memory access. For example, as a non-limiting example of a nonlinear function for deep learning, the nonlinear function may include a linear rectification (ReLU) function, a modified ReLU function, a sigmoid function, and a tanh function.

[0240] In addition, by providing a lookup table for use in the functional module, nonlinear functions can be processed with low power. Alternatively, values ​​not present in the lookup table can be interpolated by reading the two values ​​closest to the value not present in the lookup table from the lookup table and estimating the value of the nonlinear function (such as by local linear local approximation).

[0241] Figures 1 to 17The memory device 110, memory unit 120, functional module 130, input data conversion circuit 131, filter data conversion circuit 133, output data conversion circuit 135, high-speed operation processor 140, memory unit 510, input data conversion circuit 520, memory unit 610, filter data conversion circuit 620, high-speed operation processor 710, memory unit 810, output data conversion circuit 820 and MAC that perform the operations described in the present application are implemented by hardware components configured to perform the operations performed by the hardware components described in the present application. Examples of hardware components that can be used to perform the operations described in the present application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in the present application. In other examples, one or more of the hardware components that perform the operations described in the present application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a limited manner to achieve the desired result). In one example, a processor or computer includes or is connected to one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or computer can execute instructions or software (such as operating systems (OS) and one or more software applications running on OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For simplicity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but in other examples, multiple processors or computers can be used, or a processor or computer can include multiple processing elements or multiple types of processing elements or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may implement a single hardware component or two or more hardware components.The hardware components may have any one or more of different processing configurations, examples of which include: a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0242] Figures 1 to 17 The method for performing the operations described in the present application shown in the embodiment is performed by computing hardware (e.g., by one or more processors or computers), which is implemented as instructions or software as described above to perform the operations performed by the method described in the present application. For example, a single operation or two or more operations may be performed by a single processor or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller may perform a single operation or two or more operations.

[0243] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform methods as described above may be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods as described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) directly executed by one or more processors or computers. In another example, the instructions or software include high-level code executed by one or more processors or computers using an interpreter. Instructions or software may be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification, wherein the block diagrams and flow charts shown in the accompanying drawings and the corresponding descriptions in the specification disclose algorithms for performing operations performed by the hardware components and methods as described above.

[0244] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), random access memory (RAM), flash memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state disk, and any other device, wherein any other device is configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that one or more processors and computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0245] Although the present disclosure includes specific examples, it will be clear after understanding the disclosure of the present application that various changes in form and detail can be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered to be descriptive only, and not for the purpose of limitation. The description of the features or aspects in each example will be considered to be applicable to similar features or aspects in other examples. If the described techniques are performed in a different order, and / or if the components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents, appropriate results can be achieved. Therefore, the scope of the disclosure is not limited by specific embodiments, but by the claims and their equivalents, and all changes within the scope of the claims and their equivalents will be interpreted as included in the disclosure.

Claims

1. A memory device, comprising: A memory configured to store input data and filter data for a convolution operation; as well as A functional processor is configured to: respond to a read command of at least a portion of the input data and the filter data, transform at least a portion of the input data using an input transformation matrix and transform at least a portion of the filter data using a filter transformation matrix based on parameters of the convolution operation during a clock cycle corresponding to the read command, and output the corresponding transformation results as transformed data.

2. The memory device of claim 1, wherein: The function processor comprises an input data transformation circuit, the input data transformation circuit being configured to: transform the at least a portion of the input data and output a corresponding transformation result as the transformed input data, and The input data transformation circuit is configured based on the parameters of the convolution operation and the type of algorithm applied to transform the at least a portion of the input data to reduce the computational effort of the convolution operation.

3. The memory device of claim 2, wherein: Determine the input transformation matrix based on both the parameters of the convolution operation and the type of algorithm, and The input data transformation circuit is constructed to correspond to the input transformation matrix.

4. The memory device of claim 1, wherein: The function processor includes a filter data transformation circuit, the filter data transformation circuit being configured to: transform the at least a portion of the filter data and output a corresponding transformation result as transformed filter data, and The filter data transformation circuit is configured based on the parameters of the convolution operation and the type of algorithm applied to transform the at least a portion of the filter data to reduce the computational effort of the convolution operation.

5. The memory device of claim 4, wherein: determining the filter transformation matrix based on both the parameters of the convolution operation and the type of algorithm, and The filter data transformation circuit is constructed to correspond to the filter transformation matrix.

6. The memory device of claim 1, wherein: The filter data stored in the memory includes the filter data after the parameter transformation based on the convolution operation, and When the filter data is transformed filter data based on parameters of a convolution operation, the function processor is configured to output the transformed filter data without further transforming the transformed filter data.

7. The memory device of claim 1, wherein: The function processor includes a multiplier accumulator configured to perform an operation between the transformed input data and the transformed filter data.

8. The memory device of claim 1, wherein: The functional processor is configured to: in response to a write command of at least a portion of intermediate output data output by an operation between transformed input data and transformed filter data, transform the at least a portion of the intermediate output data based on a parameter of a convolution operation during a clock cycle corresponding to the write command, and output a corresponding transformation result as the transformed intermediate output data.

9. The memory device of claim 8, wherein: The intermediate output data includes the output data of the element-by-element multiplication between the transformed input data and the transformed filter data.

10. The memory device of claim 8, wherein: The function processor comprises an output data transformation circuit, the output data transformation circuit being configured to transform the at least a portion of the intermediate output data and output a corresponding transformation result as transformed intermediate output data, and The output data transformation circuit is constructed based on the parameters of the convolution operation and the type of algorithm that transforms the intermediate output data to reduce the computational effort of the convolution operation.

11. The memory device of claim 10, wherein: determining the output transformation matrix based on the parameters of the convolution operation and the type of algorithm, and The output data transformation circuit is constructed to correspond to the output transformation matrix.

12. The memory device of claim 1, wherein: Parameters of the convolution operation are determined based on any one of the size of the output data, the size of the filter data, the size of the input data, the step interval, and the padding size, or any combination of any two or more thereof.

13. The memory device of claim 1, wherein: The functional processor includes either or both of an adder and a shifter.

14. The memory device of claim 1, wherein: The algorithm that transforms the input data to reduce the computational effort of the convolution operation is the Winograd algorithm.

15. The memory device of claim 1, wherein: An algorithm that transforms the input data to reduce the computational effort of the convolution operation is the Strassen algorithm.

16. The memory device of claim 1, wherein: The function processor includes a pruning circuit configured to output a predetermined value in response to a size of the transformed input data being less than or equal to a threshold value.

17. The memory device of claim 1, wherein: The function processor is configured to calculate a value of a non-linear function used to determine whether to activate the transformed filter data.

18. The memory device of claim 1, wherein: The function processor includes an operation processor configured to perform an operation using the transformed input data and the transformed filter data.

19. The memory device of claim 18, wherein: The operation processor includes a multiplier configured to perform element-by-element multiplication using the transformed input data and the transformed filter data.

20. The memory device of claim 1, wherein: A functional processor includes two or more of the following: An input data conversion circuit configured to convert and output the at least a portion of the input data; a filter data conversion circuit configured to convert and output the at least a portion of the filter data; a multiplier configured to perform multiplication using the transformed input data and the transformed filter data; as well as The output data conversion circuit is configured to convert and output at least a part of the intermediate output data output by performing the multiplication.

21. A computing device comprising: A memory configured to store input data and filter data for a convolution operation; a function processor configured to: in response to a read command of at least a portion of the input data and the filter data, transform at least a portion of the input data using an input transformation matrix and transform at least a portion of the filter data using a filter transformation matrix based on a parameter of the convolution operation during a clock cycle corresponding to the read command, and output a corresponding transformation result as transformed data; as well as The direct memory access processor is configured to align and store the at least a portion of the data in the memory based on a connection relationship between the memory and the functional processor.

22. The computing device of claim 21, wherein: The function processor comprises an input data transformation circuit, the input data transformation circuit being configured to: transform the at least a portion of the input data and output a corresponding transformation result as the transformed input data, and The input data transformation circuit is configured based on the parameters of the convolution operation and the type of algorithm applied to transform the at least a portion of the input data to reduce the computational effort of the convolution operation.

23. The computing device of claim 22, wherein: determining the input transformation matrix based on a combination of the parameters of the convolution operation and the type of algorithm, and The input data transformation circuit is constructed to correspond to the input transformation matrix.

24. The computing device of claim 21, wherein: The function processor includes a filter data transformation circuit, the filter data transformation circuit being configured to: transform the at least a portion of the filter data and output a corresponding transformation result as transformed filter data, and The filter data transformation circuit is configured based on the parameters of the convolution operation and the type of algorithm applied to transform the at least a portion of the filter data to reduce the computational effort of the convolution operation.

25. The computing device of claim 21, further comprising: The operation processor is configured to perform an operation using the transformed input data and the transformed filter data.

26. The computing device of claim 25, wherein: The computing processor includes any one of a central processing unit, a graphics processing unit, a digital signal processor, a neural processor, and a field programmable gate array, or any combination of any two or more thereof.

27. The computing device of claim 21, wherein: The functional processor is configured to: in response to a write command of at least a portion of intermediate output data output by an operation using transformed input data and transformed filter data, transform the at least a portion of the intermediate output data based on a parameter of the convolution operation during a clock cycle corresponding to the write command, and output the corresponding transformation result as the transformed intermediate output data.

28. The computing device of claim 27, wherein: The function processor comprises an output data transformation circuit, the output data transformation circuit being configured to transform the at least a portion of the intermediate output data and output a corresponding transformation result as transformed intermediate output data, and The output data transformation circuit is constructed based on the parameters of the convolution operation and the type of algorithm that transforms the intermediate output data to reduce the computational effort of the convolution operation.

29. A processor-implemented operation method for a memory device, the operation method comprising: In response to a read command, receiving at least a portion of the input data and the filter data from the memory; as well as At least a portion of the input data is transformed using an input transformation matrix and at least a portion of the filter data is transformed using a filter transformation matrix based on parameters of the convolution operation during a clock cycle corresponding to the read command, and the corresponding transformation results are output as transformed data.

30. The operation method according to claim 29, wherein: The output steps include: determining an input transformation matrix based on a combination of parameters of the convolution operation and a type of algorithm that transforms the at least a portion of the input data to reduce the computational effort of the convolution operation; and The at least a portion of the input data is transformed based on the input transformation matrix, and the corresponding transformation result is output as the transformed input data.

31. The operation method according to claim 29, wherein: The output steps include: determining a filter transform matrix based on a combination of parameters of the convolution operation and a type of algorithm for transforming the at least a portion of the filter data to reduce the computational effort of the convolution operation; and The at least a portion of the filter data is transformed based on the filter transformation matrix, and the corresponding transformation result is output as transformed filter data.

32. The computing method according to claim 29, further comprising: In response to a write command of at least a portion of intermediate output data output by an operation between transformed input data and transformed filter data, the at least a portion of the intermediate output data is transformed based on parameters of the convolution operation during a clock cycle corresponding to the write command, and the at least a portion of the transformed intermediate output data is stored in a memory.

33. The operation method according to claim 32, wherein: The steps of storing in memory include: determining an output transformation matrix based on both the parameters of the convolution operation and the type of algorithm that transforms the at least a portion of the intermediate output data to reduce the computational effort of the convolution operation; and The at least a portion of the intermediate output data is transformed based on an output transformation matrix, and the at least a portion of the transformed intermediate output data is stored in a memory.

34. A non-transitory computer-readable storage medium storing instructions, wherein when the instructions are executed by one or more processors, the one or more processors are caused to perform the operation method of claim 29.

35. A computing device comprising: One or more processors are configured to: respond to a read command of at least a portion of input data and filter data stored in a memory including a memory function, transform at least a portion of the input data using an input transformation matrix and transform at least a portion of the filter data using a filter transformation matrix based on parameters of a convolution operation during a clock cycle corresponding to the read command, and output corresponding transformation results as transformed data.

36. The computing device of claim 35, wherein: The memory is configured to store input data and filter data.

37. The computing device of claim 36, further comprising: The direct memory access processor is configured to align and store the at least a portion of the data in the memory based on a connection relationship between the memory and the one or more processors.

38. The computing device of claim 35, wherein: The one or more processors include a multiplier accumulator configured to perform an operation between transformed input data and transformed filter data.

Images