Convolution circuit and convolution calculation method

Abstract

The application provides a convolution circuit and a convolution calculation method. The convolution circuit comprises a buffer memory and an operation circuit. The operation circuit receives input elements of an input matrix according to a memory access sequence, and judges whether each weight of a weight matrix corresponding to a filter position is in an operation range according to the coordinates of the input elements. For each weight in the operation range, the operation circuit calculates an index value of the buffer memory according to the coordinates of the input elements and the coordinates of the corresponding weight, reads a temporary storage value in the buffer memory at the index value, and adds the product of the input element and the weight to the temporary storage value. If the adding times of the temporary storage value meet a predetermined number, the temporary storage value is output. If the adding times do not meet the predetermined number, the temporary storage value is stored back to the buffer memory.

Description

Technical Field

[0001] This disclosure relates to a circuit capable of performing convolution operations. Background Technology

[0002] Convolutional neural networks (CNNs) have been used in many fields. When a CNN performs convolution operations, it needs to store two-dimensional input data in memory, which requires a significant amount of memory. How to perform convolution operations with less memory is a problem that those skilled in the art have sought to solve. Summary of the Invention

[0003] This disclosure provides an embodiment of a convolution circuit, including a buffer memory and an arithmetic circuit electrically connected to each other. The arithmetic circuit receives input elements of an input matrix according to the memory access order, wherein multiple weights of a weight matrix are stored in the arithmetic circuit. The arithmetic circuit determines whether the filter position corresponding to each weight is within the computational range based on the coordinates of the input elements. For each weight within the computational range, the arithmetic circuit calculates the index value of the buffer memory based on the coordinates of the input element and the coordinates of the corresponding weight, reads a temporary value at the index value from the buffer memory, multiplies the input element by the weight to obtain a product, and accumulates the product to the temporary value. If the accumulation count of the temporary value meets a predetermined number, the arithmetic circuit outputs the temporary value as an output element of the output matrix and resets the accumulation count. If the accumulation count does not meet the predetermined number, the arithmetic circuit stores the temporary value back in the buffer memory.

[0004] From another perspective, embodiments of this disclosure propose a convolution calculation method applicable to arithmetic circuits. This convolution calculation method includes: receiving input elements of an input matrix according to memory access order; determining whether the filter position corresponding to each weight in a weight matrix is ​​within the computation range based on the coordinates of the input elements; for each weight within the computation range, calculating the index value of a buffer memory based on the coordinates of the input elements and the coordinates of the corresponding weight; reading a temporary value located at the index value from the buffer memory, multiplying the input elements by the weights to obtain a product, and accumulating the product to the temporary value; if the accumulation count of the temporary value meets a predetermined number, outputting the temporary value as an output element of the output matrix and resetting the accumulation count; and if the accumulation count does not meet the predetermined number, storing the temporary value back into the buffer memory.

[0005] In the convolution circuits and convolution calculation methods described above, less memory can be used to perform convolution operations.

[0006] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings. Attached Figure Description

[0007] Figure 1 This is a schematic diagram illustrating a convolution circuit according to one embodiment.

[0008] Figure 2 This is a schematic diagram showing the location of the filter according to one embodiment.

[0009] Figure 3 This is a schematic diagram illustrating the output matrix and buffer memory size according to one embodiment.

[0010] Figure 4 Table 400 is shown according to one embodiment, which records the multiplication required for each output element.

[0011] Figure 5A and Figure 5B Tables 500A and 500B are shown according to one embodiment, recording the relevant calculation information corresponding to each input element.

[0012] Figure 6 This is a schematic diagram illustrating the operation of a convolutional circuit in a trusted environment, according to one embodiment.

[0013] Figure 7 This is an example illustrating a convolution circuit with encryption and decryption capabilities.

[0014] Figure 8 This is a flowchart illustrating a convolution calculation method according to an embodiment.

[0015] Symbol Explanation

[0016] 110: Input matrix

[0017] 111: Memory Access Order

[0018] 120: Convolutional Circuit

[0019] 130: Buffer memory

[0020] 131: Temporary value

[0021] 140: Operational circuit

[0022] 141: Multiplier

[0023] 142, 143: Adders

[0024] 145: Step Quantity

[0025] 146: Weight

[0026] 147: Offset

[0027] 150: Output matrix

[0028] A[x,y]: Input elements

[0029] C[m,n]: Output element

[0030] 210: Filter

[0031] 220: Filter

[0032] 230: Fill Range

[0033] X,I,M: Number of rows

[0034] Y, J, N: Number of columns

[0035] P: Filling volume

[0036] 310: Diagonal area

[0037] 400, 500A, 500B: Tables

[0038] 610: Trust Environment

[0039] 620: Distrust of the environment

[0040] 630: Shared Storage

[0041] 640: Other programs or circuits

[0042] 710: Decryption Circuit

[0043] 720: Encryption Circuit

[0044] 801~809: Steps Detailed Implementation

[0045] In this disclosure, a two-dimensional input matrix is ​​converted into one-dimensional data. During convolution operations, elements in the one-dimensional data are processed according to the memory access order, and all necessary multiplication operations are performed for each element. Once processing is complete, the next element can be read. Since previous elements are not required, this invention does not require storing the entire input matrix in memory.

[0046] Figure 1 This is a schematic diagram illustrating a convolution circuit according to one embodiment. Please refer to... Figure 1The convolution circuit 120 includes a buffer memory 130 and an arithmetic circuit 140 electrically connected to each other. The buffer memory 130 may be a random access memory or flash memory, etc. The buffer memory 130 is used to store multiple temporary values ​​131, which are initialized to zero. The arithmetic circuit 140 is used to perform a convolution calculation method. The arithmetic circuit 140 includes a multiplier 141, an adder 142, an adder 143, and other logic circuits (not shown). The arithmetic circuit 140 also stores multiple weights 146 and biases 147 of a weight matrix (also called a filter). These weights 146 and biases 147 are used to perform convolution operations; those skilled in the art will understand the details of the convolution operation, which will not be elaborated here.

[0047] First, the arithmetic circuit 140 receives an input element from the input matrix 110 according to the memory access order 111. Figure 1 The process proceeds from left to right (reading elements from one column first), and then from top to bottom (reading elements from the next column). The memory access order 111 refers to accessing elements in the input matrix 110 based on consecutive memory addresses. During convolution, the filters move continuously, and the weights in the filters are multiplied by the corresponding input elements. That is, one input element corresponds to multiple filter positions. For example, Figure 2 This is a schematic diagram illustrating the filter location according to one embodiment. Here, it is assumed that the input matrix 110 is matrix A, with a size of... The filter can be viewed as a weight matrix B, with a size of Where X, Y, I, and J are positive integers, with I and X also called the number of columns, and Y and J also called the number of rows. Positive integer I is less than positive integer X, and positive integer J is less than positive integer Y. A[x,y] represents the input element located at coordinates (x,y) in input matrix 110, that is, the element in the x-th column and y-th row of input matrix 110. B[i,j] represents the weight located at coordinates (i,j) in the weight matrix. Figure 2 In the embodiment, the filter size is Therefore, the filter includes nine weights: B[0,0], B[0,1], ..., B[2,2]. When the center point of the filter moves to coordinates (x-1, y-1) (as shown in filter 210), weight B[2,2] is multiplied by the input element A[x,y]. When the center point of the filter moves to coordinates (x+1, y+1) (as shown in filter 220), weight B[0,0] is multiplied by the input element A[x,y]. However, when the input element A[x,y] is located at the four edges of the input matrix 110, the filter may exceed the range of the input matrix 110, so not all weights B[i,j] will be multiplied by the input element A[x,y]. For example, when the coordinates (x,y) are (0,0), weight B[2,2] will not be multiplied by the input element A[x,y]. Therefore, for input element A[x,y], it is necessary to first determine whether the filter position corresponding to each weight B[i,j] is within the computation range based on the coordinates (x,y). If the input matrix 110 is not padded, then the computation range is the range of the input matrix 110. If the input matrix 110 is padded, for example in... Figure 2 If all four sides of the input matrix 110 are filled with P / 2 = 1 pixel (or P / 2 = 2 pixels in other embodiments), then the calculation range is the range of the input matrix 110 plus the annular filling range 230 (with a width of P / 2) surrounding the input matrix 110. In this embodiment, the filling range 230 is symmetrical, but in other embodiments, the filling range can also be asymmetrical.

[0048] More specifically, subtracting the weight coordinates (i,j) from the input element's coordinates (x,y) yields coordinates (xi,yj), which is the top-left corner of the filter. This allows us to determine if xi and yj are less than 0, thus indicating whether the corresponding filter position exceeds the first operational boundary (left or top boundary). Furthermore, adding the filter size (I,J) to coordinates (xi,yj) yields coordinates (x-i+I, y-j+J). This allows us to determine if x-i+I is greater than a positive integer X and y-j+J is greater than a positive integer Y, thus indicating whether the corresponding filter position exceeds the second operational boundary (right or bottom boundary). Depending on the padding method, the above judgments can be expressed as either mathematical formula 1 or mathematical formula 2.

[0049] [Mathematical Expression 1]

[0050] [Mathematical Expression 2]

[0051] Formula 1 applies to the padding method provided by the CAFFE® machine learning framework, while Formula 2 applies to the padding method provided by the TensorFlow® machine learning framework. If no padding is performed, the positive integer P mentioned above can be set to zero. For the input element A[x,y], it is determined whether all coordinates (i,j) satisfy either Formula 1 or Formula 2. If they do, it means that the corresponding filter position is within the calculation range.

[0052] Please refer to Figure 1 For each weight 146 within the computational range, multiplier 141 multiplies the input element A[x,y] with weight 146 to obtain a product. This product is accumulated in the corresponding temporary value 131 in the buffer memory 130. The following explains how to find the corresponding temporary value 131. First, the coordinates of the output elements need to be calculated. Here, it is assumed that the output matrix 150 is represented as matrix C with a size of... Where M and N are positive integers, M is called the number of columns, and N is called the number of rows. , C[m,n] represents the output element located at coordinates (m,n) in the output matrix 150. Here, the coordinates (m,n) of the output element can be calculated by subtracting the coordinates (i,j) of the corresponding weight from the coordinates (x,y) of the input element. For different filling methods, the coordinates (m,n) can be calculated as shown in Equation 3 or Equation 4 below.

[0053] [Mathematical Expression 3]

[0054] [Mathematical Expression 4]

[0055] Formula 3 applies to the padding method provided by the CAFFE® machine learning framework, while Formula 4 applies to the padding method provided by the TensorFlow® machine learning framework or no padding.

[0056] The required size of buffer memory 130 will be explained next. Figure 3 This is a schematic diagram illustrating the output matrix and buffer memory size according to one embodiment. Please refer to... Figure 3 For simplicity, we will use the scenario described in Equation 4 above. For an output element C[m,n], the required calculations include A[m,n]. B[0,0]、A[m,n-1] B[0,1]、…、A[m+I-1,n+J-1] B[I-1,J-1]. That is, before the input element A[m+I-1,n+J-1] is read, the result of the multiplication must first be stored in the buffer memory 130. Only after the input element A[m+I-1,n+J-1] is read can the calculation of the output element C[m,n] be completed. Since the memory access order is from left to right and then from top to bottom, a total of [number missing] steps are required. A temporary value, such as Figure 3 The number of elements covered by the slanted region 310, in some embodiments the size of the buffer memory 130 is... This requires less memory space than existing technologies that need to store the entire input matrix.

[0057] For each weight B[i,j] within the computational range, the index value of the buffer memory 130 can be calculated based on the coordinates (x,y) of the input element, i.e., the coordinates (i,j) of the corresponding weight, as shown in the following mathematical formula 5.

[0058] [Mathematical Expression 5]

[0059] in Here, "P mod Q" represents the remainder when P is divided by Q, where Q is the index value. Please refer to [reference needed]. Figure 1 Next, the operation circuit 140 reads the temporary value 131 located at index k, hereinafter referred to as I[k], and adds the product of the input element A[x,y] and the weight B[i,j] to the temporary value I[k], as shown in the following mathematical formula 6.

[0060] [Mathematical Expression 6]

[0061] Next, it is determined whether the number of times the temporary value I[k] is accumulated meets a predetermined number, which is the same as the number of all weights in the filter. If the number of accumulations does not meet the predetermined number, it means that all multiplications have not been completed, so the temporary value I[k] is stored back into the buffer memory 130. If the number of accumulations meets the predetermined number, it means that all the required multiplications have been completed, so the temporary value I[k] can be output. The index value k and the number of accumulations of the temporary value I[k] are reset, and the memory space located at the index value k is used by subsequent output elements. When outputting the temporary value I[k], the temporary value I[k] also needs to be added with an offset of 147. Next, the coordinates (m,n) are adjusted according to the size of the step size 145, specifically, it is determined whether the conditions of the following mathematical formula 7 are met.

[0062] [Mathematical Expression 7]

[0063] in The step size is 145. If the above conditions are met, the coordinates of the output element are adjusted to (p, q), where... The calculation result of adder 143 is output as C[p,q]. If the step size 145 is equal to 1, then there is no need to modify the coordinates (m,n) of the output element, and the calculation result of adder 143 can be used as the output element C[m,n]. If the step size 145 is greater than 1 and the above conditions are not met, then no output element is generated.

[0064] Figure 4 Table 400, as shown in one embodiment, records the multiplication required for each output element. Please refer to... Figure 4 In this example, the size of the input matrix A is (X=Y=5), the size of the weight matrix B is (I=J=2), the size of the output matrix C is (M=N=4), the fill amount P is 0, the step size is 1, and the size of the buffer memory 130 is 6. For example, the calculation of the output element C[0,0] is shown in the following mathematical formula 8, and so on for other output elements.

[0065] [Mathematical Expression 8]

[0066] Each output element C[m,n] requires 4 multiplications, and must be temporarily stored in buffer memory 130 before the 4 multiplications are completed. Figure 5A and Figure 5BTables 500A and 500B are shown in one embodiment, recording the relevant calculation information corresponding to each input element. First, input A[0,0] is given, and then the judgment in Mathematical Formula 1 above is performed. Only the filter position corresponding to weight B[0,0] is within the calculation range. Therefore, input element A[0,0] and weight B[0,0] are multiplied, and the index value k=0 is calculated according to Mathematical Formula 5. The multiplication result is stored in temporary value I[0]. Since the number of accumulations is less than 4, no output element C[m,n] is generated. Next, input element A[0,1] is received, and the judgment in Mathematical Formula 1 above is performed. The filter positions corresponding to weight B[0,0] and weight [0,1] are within the calculation range. Therefore, the corresponding multiplication is performed and stored in the corresponding temporary value I[k]. The same process is repeated for input elements A[0,2]…A[1,0]. When input element A[1,1] is received, the filter positions corresponding to weights B[0,0], B[0,1], B[1,0], and B[1,1] are within the computation range. After the relevant multiplication, the accumulated count of the temporary value I[0] is equal to 4, so the output temporary value I[0] is used as the output element C[0,0]. When input element A[1,2] is received next, the temporary value I[0] and the corresponding accumulated count have been reset and can be reused. It is worth noting that the output elements are generated in the order of C[0,0], C[0,1], C[0,2], ..., which is the same as the memory access order of the output matrix.

[0067] Based on the convolution circuit described above, a two-dimensional input matrix can be converted into one-dimensional data, and elements in the one-dimensional data can be read according to the memory access order. After the relevant calculations are performed, the input elements are no longer used and do not need to be stored in memory, thus saving memory space. In some embodiments, such a convolution circuit can be applied in a trusted environment, such as ARM®'s Trust Execution Environment (TEE). Figure 6 This is a schematic diagram illustrating the operation of a convolutional circuit in a trusted environment, according to one embodiment. Please refer to... Figure 6This can be viewed from the perspective of hardware, software, or firmware architecture, and this disclosure is not limited thereto. The system includes a trusted environment 610 and an untrusted environment 620, which can transfer data between them via shared memory 630. The convolution circuit 120 is located in the trusted environment 610, while other programs or circuits 640 are located in the untrusted environment 620. The other programs or circuits 640 can be other operations of the convolutional neural network, any image, speech, or text processing programs or circuits, and this disclosure is not limited thereto. In some applications, if convolution operations are required on sensitive data, the convolution operations will be performed in the trusted environment 610. However, it is not suitable to perform too much work in the trusted environment 610, such as consuming too much processor time or too much memory space, so the convolution circuit 120 can be used. When the convolution circuit 120 is running, the other programs or circuits 640 store input elements in the shared memory 630, the convolution circuit 120 receives input elements from the shared memory 630, and stores the calculated output elements in the shared memory 630. This reduces the resources consumed in the trust environment 610. Furthermore, this also allows for the fragmentation of convolution computation. As the convolution circuit 120 processes one input element at a time, the resource consumption time in the trust environment 610 decreases, while the number of processing times increases, which aligns with the usage characteristics of the trust environment 610.

[0068] In some embodiments, the convolution circuit 120 can also be used in conjunction with a decryption circuit and an encryption circuit. Figure 7 This is an example illustrating a convolutional circuit with encryption and decryption capabilities. Please refer to... Figure 7 The convolution circuit 120 includes a buffer memory 130, an arithmetic circuit 140, a decryption circuit 710, and an encryption circuit 720. The decryption circuit 710 decrypts the input data to obtain the input elements, which are then transmitted to the arithmetic circuit 140. The encryption circuit 720 encrypts the resulting output elements. Current methods involve decrypting the entire two-dimensional input matrix and storing it in memory before performing convolution operations on the input matrix. However, this approach exposes the decrypted data in memory. Figure 7 This embodiment can avoid decrypting all input matrices at once, reducing memory usage and avoiding exposing the decrypted data.

[0069] exist Figure 2 In one embodiment, the arithmetic circuit 140 includes a set of multipliers 141, adders 142 and 143. However, in other embodiments, the arithmetic circuit 140 may also include more adders and multipliers, and can simultaneously receive multiple input elements and perform parallel processing.

[0070] Figure 8 This is a flowchart illustrating a convolution calculation method according to an embodiment. Please refer to it. Figure 8The process begins at step 801. In step 802, input elements of the input matrix are received according to the memory access order. In step 803, the filter position corresponding to each weight in the weight matrix is ​​determined to be within the computational range based on the coordinates of the input elements. In step 804, for each weight within the computational range, the index value of the buffer memory is calculated based on the coordinates of the input elements and the coordinates of the corresponding weight. In step 805, a temporary value at the index value is read from the buffer memory, the input element is multiplied by the weight to obtain a product, and the product is accumulated to the temporary value. In step 806, it is determined whether the accumulation count of the temporary value meets the predetermined count. If the result of step 806 is yes, in step 807, the temporary value is output as the output element of the output matrix, and the accumulation count is reset. If the result of step 806 is no, in step 808, the temporary value is stored back in the buffer memory. This process ends in step 809. Figure 8 Each step has been explained in detail above and will not be repeated here. It is worth noting that... Figure 8 Each step can be implemented as multiple program codes or circuits, and this invention is not limited thereto. Furthermore, Figure 8 The method can be used in conjunction with the above embodiments or alone; in other words, Figure 8 Other steps can also be added between the various steps.

[0071] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some changes and modifications without departing from the concept and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A convolution circuit, comprising: A buffer memory; as well as An arithmetic circuit, electrically connected to the buffer memory, is used to receive an input element of an input matrix according to a memory access order, wherein multiple weights of a weight matrix are stored in the arithmetic circuit. The computational circuit is used to determine whether the position of a filter corresponding to each of the multiple weights is within a computational range based on the coordinates of the input element. For each of the multiple weights within the computational range, the computational circuit calculates an index value of the buffer memory based on the coordinates of the input element and the coordinates of the corresponding weight, reads a temporary value located at the index value from the buffer memory, multiplies the input element by the weight to obtain a product, and adds the product to the temporary value. If the number of times the temporary value is accumulated meets a predetermined number, the arithmetic circuit outputs the temporary value as an output element of an output matrix and resets the number of times the value is accumulated. If the number of accumulations does not meet the predetermined number, the arithmetic circuit stores the temporary value back into the buffer memory.

2. The convolution circuit of claim 1, wherein the operation of the computation circuit in determining whether the filter position of each of the plurality of weights is within the computation range includes: Subtract the corresponding weight's coordinate from the coordinate of the input element to determine whether the filter's position exceeds the first operation boundary; as well as Subtract the corresponding weight's coordinate from the coordinate of the input element and add the filter size to determine whether the filter position exceeds the second operation boundary.

3. The convolution circuit as described in claim 2, wherein the operational range is the range of the input matrix plus a padding range.

4. The convolution circuit of claim 1, wherein for each of the plurality of weights within the computational range, the computational circuit subtracts the coordinate of the corresponding weight from the coordinate of the input element to calculate the coordinate of the output element.

5. The convolution circuit as described in claim 4, wherein the coordinates of the output element are (m, n), and the operational circuit calculates the index value according to the following mathematical formula. k = (N*m+n)mod((I-1)*N+J) Where k is the index value, N is the number of rows in the output matrix, I is the number of columns in the weight matrix, J is the number of rows in the weight matrix, and "P mod Q" represents the remainder obtained after dividing P by Q.

6. The convolution circuit of claim 5, wherein the operational circuit determines whether the following conditions are met. stride≥2,m mod stride=0, and n mod stride=0 Where stride is the step size. If the condition is met, the operational circuit adjusts the coordinates of the output element to (p, q), where p = m / stride and q = n / stride.

7. The convolution circuit of claim 1, wherein the predetermined number of times is the same as the number of the plurality of weights.

8. The convolution circuit of claim 1, wherein when outputting the temporary value, the arithmetic circuit also adds an offset to the temporary value.

9. The convolution circuit of claim 1, wherein the buffer memory and the computation circuit are in a trusted environment, and the computation circuit receives the input element through a shared memory and stores the output element in the shared memory.

10. A convolution calculation method, applicable to an operational circuit, the convolution calculation method comprising: An input element of an input matrix is ​​received according to a memory access sequence. Determine whether the position of a filter corresponding to each of the multiple weights in a weight matrix is ​​within the computation range based on the coordinates of the input element. For each of the plurality of weights within the computational range, an index value of a buffer memory is calculated based on the coordinates of the input element and the coordinates of the corresponding weight. Read a temporary value at the index from the buffer memory, multiply the input element by the weight to obtain a product, and add the product to the temporary value; If the number of times the temporary value is accumulated meets a predetermined number, output the temporary value as an output element of an output matrix and reset the number of times the value is accumulated. as well as If the number of accumulations does not meet the predetermined number, the temporary value is stored back into the buffer memory.

Images