Convolution operation device

Abstract

This invention provides a convolution operation device, including a first memory, a second memory, a third memory, a first multiply-accumulate circuit, a second multiply-accumulate circuit, and a routing and shift register circuit. Different elements of the same matrix are stored in different memories. The first and second multiply-accumulate circuits retrieve the convolution kernel from the first memory. In a first period, the routing and shift register circuit transfers the first element of the matrix from the second memory to the first multiply-accumulate circuit, and transfers the second element of the matrix from the third memory to the second multiply-accumulate circuit. In a second period, the routing and shift register circuit transfers the second element of the third memory to the first multiply-accumulate circuit.

Description

[Technical Field]

[0001] The present invention relates to an electronic circuit, and more particularly to a convolution operation device. [Previous Technology]

[0002] Convolution is one of the most common operations in neural network models. When using matrices for convolution, to improve the efficiency of the processing unit, the elements of the input matrix are copied multiple times to different memory locations corresponding to different multiply accumulate (MAC) operators. It is conceivable that this redundant storage of element data will affect memory utilization efficiency. [Summary of the Invention]

[0003] The present invention provides a convolution operation device to avoid redundant storage of the same elements of a matrix in different memories.

[0004] In one embodiment of the present invention, the above-described convolution operation device includes a first memory, a second memory, a third memory, a first multiply-accumulate circuit, a second multiply-accumulate circuit, and a routing and shift register circuit. The first memory is used to store the convolution kernel. The second memory is used to store the first element of the matrix. The third memory is used to store the second element of the matrix. The first multiply-accumulate circuit and the second multiply-accumulate circuit are coupled to the first memory to retrieve the convolution kernel. The routing and shift register circuit is coupled to the second memory, the third memory, the first multiply-accumulate circuit, and the second multiply-accumulate circuit. During a first period, the routing and shift register circuit transfers the first element of the second memory to the first multiply-accumulate circuit, and the routing and shift register circuit transfers the second element of the third memory to the second multiply-accumulate circuit. During a second period, the routing and shift register circuit transfers the second element of the third memory to the first multiply-accumulate circuit.

[0005] Based on the above, different parts of the matrix are stored in different memories to avoid redundant storage of element data. When a multiply-accumulate circuit's target element is not in its corresponding memory, the routing and shift register circuit can retrieve the target element from the memory corresponding to other multiply-accumulate circuits and assign it to the multiply-accumulate circuit. Therefore, the convolution operation device provides a hardware computing architecture that improves memory efficiency.

[0006] In order to make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below, and detailed descriptions are given in conjunction with the accompanying drawings.

Implementation Method

[0008] The term "coupled (or connected)" as used throughout the entire specification (including the claims) may refer to any direct or indirect means of connection. For example, if the text describes a first device coupled (or connected) to a second device, it should be interpreted as the first device being directly connected to the second device, or the first device being indirectly connected to the second device through other devices or some means of connection. The terms "first," "second," etc., used throughout the entire specification (including the claims) are used to name elements or distinguish different embodiments or scopes, and are not intended to limit the upper or lower limit of the number of elements, nor to limit the order of elements. Furthermore, wherever possible, elements / components / steps using the same reference numerals in the drawings and embodiments represent the same or similar parts. Elements / components / steps using the same reference numerals or the same terms in different embodiments may be referred to mutually in the relevant descriptions.

[0009] Figure 1 is a schematic circuit block diagram of a convolution operation device 200 according to an embodiment of the present invention. Based on the control of the host device 100, the convolution operation device 200 can perform various neural network model operations (e.g., convolution operations). In the embodiment shown in Figure 1, the convolution operation device 200 includes a memory 210 and a convolution operation circuit 220. The host device 100 can store the convolution kernels of a trained neural network model in the memory 210 for use by the convolution operation circuit 220. The convolution operation circuit 220 is coupled to the memory 210. Based on the contents of the memory 210, the convolution operation circuit 220 can perform convolution operations to obtain a result matrix.

[0010] Figure 2 is a schematic diagram illustrating a convolution operation according to an application example. In the embodiment shown in Figure 2, it is assumed that the elements of matrix MX2 include X00, X01, X02, and X03, the elements of convolution kernel MZ2 include Z00, Z01, Z02, and Z03, the stride parameter of the convolution operation is 1, and the padding parameter of the convolution operation is 3 (that is, matrix MX2 is additionally padded with 3 padding elements P). The specific value of the padding element P can be defined according to the actual application. For example, the padding element P can be 0 or other real numbers.

[0011] Referring to Figures 1 and 2, memory 210 is used to store matrix MX2 and convolution kernel MZ2. Based on the contents of memory 210, convolution operation circuit 220 can perform convolution operation to obtain operation result matrix MY2. In the embodiment shown in Figure 2, because matrix MX2 is additionally padded with 3 padding elements P, the operation result matrix MY2 is a 1*4 matrix (its elements include Y00, Y01, Y02 and Y03, as shown in Figure 2).

[0012] FIG3 is a circuit block diagram of a convolution operation device 300 according to an embodiment. The convolution operation device 300 shown in FIG3 includes memory 311, memory 312, memory 313, memory 314, memory 315 and convolution operation circuit 320. The convolution operation circuit 320 shown in FIG3 can be one of many embodiments of the convolution operation circuit 220 shown in FIG1. ​​The memory 311-315 shown in FIG3 can be one of many embodiments of the memory 210 shown in FIG1. ​​In the embodiment shown in FIG3, the convolution operation circuit 320 includes 4 multiply accumulate (MAC) operators and 8 registers (REG). This embodiment does not limit the specific implementation of the MAC operators and registers. For example, the MAC operators can be conventional MAC operators or other multiply accumulate circuits, and the registers can be conventional registers or other data register circuits.

[0013] Referring to Figures 2 and 3, the elements Z00, Z01, Z02, and Z03 of the convolution kernel MZ2 are stored in memory 311. To improve the efficiency of the convolution operation, in the embodiment shown in Figure 3, multiple elements of matrix MX2 are copied multiple times to different memories 312-315 corresponding to different MAC processors. As shown in Figure 3, the elements X00, X01, X02, and X03 of matrix MX2 are stored in memory 312; the elements X01, X02, and X03 of matrix MX2 and one padding element P are stored in memory 313; the elements X02 and X03 of matrix MX2 and two padding elements P are stored in memory 314; and the element X03 of matrix MX2 and three padding elements P are stored in memory 315.

[0014] During the first period, memory units 312-315 each provide the first elements X00, X01, X02, and X03 to different MAC processors, and memory unit 311 provides the first element Z00 to these four MAC processors. Therefore, after the MAC operation in the first period, element Y00 is X00*Z00, element Y01 is X01*Z00, element Y02 is X02*Z00, and element Y03 is X03*Z00.

[0015] During the second period, memory units 312-315 each provide the second elements X01, X02, X03 and P to different MAC operators, and memory unit 311 provides the second element Z01 to these four MAC operators. Therefore, after the MAC operation in the second period, element Y00 is X00*Z00 + X01*Z01, element Y01 is X01*Z00 + X02*Z01, element Y02 is X02*Z00 + X03*Z01, and element Y03 is X03*Z00 + p*Z01.

[0016] During the third period, memory units 312-315 each provide the third elements X02, X03, P, and P to different MAC operators, and memory unit 311 provides the third element Z02 to these four MAC operators. Therefore, after the MAC operation is completed in the third period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02, element Y01 is X01*Z00 + X02*Z01 + X03*Z02, element Y02 is X02*Z00 + X03*Z01 + p*Z02, and element Y03 is X03*Z00 + p*Z01 + p*Z02.

[0017] During the fourth period, memory units 312-315 each provide the fourth element X03, P, P, and P to different MAC operators, and memory unit 311 provides the fourth element Z03 to these four MAC operators. Therefore, after the MAC operation is completed in the fourth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X03*Z03, element Y01 is X01*Z00 + X02*Z01 + X03*Z02 + p*Z03, element Y02 is X02*Z00 + X03*Z01 + p*Z02 + p*Z03, and element Y03 is X03*Z00 + p*Z01 + p*Z02 + p*Z03.

[0018] To improve the efficiency of convolution operations, as shown in Figure 3, multiple elements of matrix MX2 are copied multiple times to different memory locations 312-315 corresponding to different MAC processors. For example, element X03 of matrix MX2 is copied to memory locations 312-315. It is conceivable that copying the same element data to different memory locations will affect memory utilization efficiency. The following embodiments will illustrate a convolution operation device with efficient memory storage.

[0019] Figure 4 is a circuit block diagram of a convolution operation device 400 according to an embodiment of the present invention. The convolution operation device 400 shown in Figure 4 includes memory 411, memory 412, memory 413, memory 414, memory 415 and convolution operation circuit 420. Memory 411 to 415 shown in Figure 4 can be one of many embodiments of memory 210 shown in Figure 1. Referring to Figures 2 and 4, the elements Z00, Z01, Z02 and Z03 of the convolution kernel MZ2 are stored in memory 411. The element X00 (first part) of matrix MX2 is stored in memory 412, the element X01 (second part) of matrix MX2 is stored in memory 413, the element X02 (third part) of matrix MX2 is stored in memory 414, and the element X03 (fourth part) of matrix MX2 is stored in memory 415. The partial matrices stored in different memories are mutually exclusive. That is, no element of matrix MX2 will be placed repeatedly in different memory locations. Different parts of matrix MX2 are stored in different memory locations 412-415 to avoid redundant storage of element data.

[0020] The convolution operation circuit 420 shown in Figure 4 can be one of many implementation examples of the convolution operation circuit 220 shown in Figure 1. In the embodiment shown in Figure 4, the convolution operation circuit 420 includes a routing and shift register circuit 421 and multiple multiply-accumulate circuits. The specific number of multiply-accumulate circuits can be determined according to the actual design. In the embodiment shown in Figure 4, the convolution operation circuit 420 includes four multiply-accumulate circuits 422_0, 422_1, 422_2 and 422_3. Multiply-accumulate circuits 422_0 to 422_3 are coupled to memory 411 to access the convolution kernel MZ2. Multiply-accumulate circuit 422_0 corresponds to memory 412, multiply-accumulate circuit 422_1 corresponds to memory 413, multiply-accumulate circuit 422_2 corresponds to memory 414, and multiply-accumulate circuit 422_3 corresponds to memory 415.

[0021] The multiply-accumulate circuits 422_0 to 422_3 have similar circuit structures. Taking the multiply-accumulate circuit 422_0 as an example, the multiply-accumulate circuit 422_0 includes one multiply-accumulate (MAC) arithmetic unit and one temporary register (REG). In the multiply-accumulate circuit 422_0, the input terminal of the temporary register is coupled to the memory 411, the first input terminal of the MAC arithmetic unit is coupled to the output terminal of the temporary register, and the second input terminal of the MAC arithmetic unit is coupled to the routing and shifting temporary register circuit 421. The MAC arithmetic units in the multiply-accumulate circuits 422_0 to 422_3 can be deduced by referring to the relevant description of the MAC arithmetic unit shown in Figure 3, and therefore will not be described again.

[0022] The routing and shift register circuit 421 is coupled to memory modules 412-415 and multiply-accumulate circuits 422_0-422_3. During the first period, the routing and shift register circuit 421 transfers element X00 of memory module 412 to multiply-accumulate circuit 422_0, transfers element X01 of memory module 413 to multiply-accumulate circuit 422_1, transfers element X02 of memory module 414 to multiply-accumulate circuit 422_2, transfers element X03 of memory module 415 to multiply-accumulate circuit 422_3, and memory module 411 provides the first element Z00 to multiply-accumulate circuits 422_0-422_3. Therefore, after the MAC operation in the first period, element Y00 is X00*Z00, element Y01 is X01*Z00, element Y02 is X02*Z00, and element Y03 is X03*Z00.

[0023] During the second period, memory 411 provides the second element Z01 to multiply-accumulate circuits 422_0 to 422_3; routing and shift register circuit 421 transfers element X01 of memory 413 to multiply-accumulate circuit 422_0; routing and shift register circuit 421 transfers element X02 of memory 414 to multiply-accumulate circuit 422_1; routing and shift register circuit 421 transfers element X03 of memory 415 to multiply-accumulate circuit 422_2; and routing and shift register circuit 421 transfers fill element P to multiply-accumulate circuit 422_3. The specific value of fill element P can be defined according to the actual application. For example, fill element P can be 0 or other real numbers. Therefore, after the MAC operation in the second period, element Y00 is X00*Z00 + X01*Z01, element Y01 is X01*Z00 + X02*Z01, element Y02 is X02*Z00 + X03*Z01, and element Y03 is X03*Z00 + p*Z01.

[0024] During the third period, memory 411 provides the third element Z02 to multiply-accumulate circuits 422_0 to 422_3, routing and shift register circuit 421 transfers element X02 of memory 414 to multiply-accumulate circuit 422_0, routing and shift register circuit 421 transfers element X03 of memory 415 to multiply-accumulate circuit 422_1, and routing and shift register circuit 421 transfers fill element P to multiply-accumulate circuits 422_2 and 422_3. Therefore, after the MAC operation in the third period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02, element Y01 is X01*Z00 + X02*Z01 + X03*Z02, element Y02 is X02*Z00 + X03*Z01 + p*Z02, and element Y03 is X03*Z00 + p*Z01 + p*Z02.

[0025] During the fourth period, memory 411 provides the fourth element Z03 to multiply-accumulate circuits 422_0 to 422_3, routing and shift register circuit 421 transfers element X03 of memory 415 to multiply-accumulate circuit 422_0, and routing and shift register circuit 421 transfers fill element P to multiply-accumulate circuits 422_1, 422_2 and 422_3. Therefore, after the MAC operation in the fourth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X03*Z03, element Y01 is X01*Z00 + X02*Z01 + X03*Z02 + p*Z03, element Y02 is X02*Z00 + X03*Z01 + p*Z02 + p*Z03, and element Y03 is X03*Z00 + p*Z01 + p*Z02 + p*Z03.

[0026] In summary, different parts of matrix MX2 are stored in different memories 412-415 to avoid redundant storage of element data. When a multiply-accumulate circuit (e.g., 422_0) needs a target element (e.g., X01) that is not in its corresponding memory (e.g., 412), the routing and shift register circuit 421 can retrieve the target element from the memory (e.g., 413) corresponding to another multiply-accumulate circuit (e.g., 422_1) and assign it to the multiply-accumulate circuit (e.g., 422_0). Therefore, the convolution operation device 400 provides a hardware computing architecture that improves memory efficiency.

[0027] This embodiment does not limit the specific implementation of the routing and shift register circuit 421. For example, in the embodiment shown in FIG4, the routing and shift register circuit 421 includes multiplexer MUX40, register REG40, multiplexer MUX41, register REG41, multiplexer MUX42, register REG42, multiplexer MUX43, and register REG43. This embodiment does not limit the specific implementation of multiplexers MUX40-MUX43 and registers REG40-REG43. For example, multiplexers MUX40-MUX43 can be conventional multiplexers or other data routing circuits, and registers REG40-REG43 can be conventional registers or other data temporary storage circuits.

[0028] One input of multiplexer MUX40 is coupled to memory 412. The input of register REG40 is coupled to the output of multiplexer MUX40. The output of register REG40 is coupled to multiply-accumulate circuit 422_0. One input of multiplexer MUX41 is coupled to memory 413. The input of register REG41 is coupled to the output of multiplexer MUX41. The output of register REG41 is coupled to the other input of multiplexer MUX40 and multiply-accumulate circuit 422_1. One input of multiplexer MUX42 is coupled to memory 414. The input of register REG42 is coupled to the output of multiplexer MUX42. The output of register REG42 is coupled to the other input of multiplexer MUX41 and multiply-accumulate circuit 422_2. One input of multiplexer MUX43 is coupled to memory 415. The input of register REG43 is coupled to the output of multiplexer MUX43. The output of register REG43 is coupled to another input of multiplexer MUX42 and multiply-accumulate circuit 422_3. The other input of multiplexer MUX43 receives fill element P.

[0029] During the first period, multiplexer MUX40 transfers element X00 of memory 412 to register REG40, multiplexer MUX41 transfers element X01 of memory 413 to register REG41, multiplexer MUX42 transfers element X02 of memory 414 to register REG42, and multiplexer MUX43 transfers element X03 of memory 415 to register REG43. During the second period, multiplexer MUX40 transfers the output (element X01) of register REG41 to register REG40, multiplexer MUX41 transfers the output (element X02) of register REG42 to register REG41, multiplexer MUX42 transfers the output (element X03) of register REG43 to register REG42, and multiplexer MUX43 transfers fill element P to register REG43. During the third period, multiplexer MUX40 transmits the output of register REG41 (element X02) to register REG40, multiplexer MUX41 transmits the output of register REG42 (element X03) to register REG41, multiplexer MUX42 transmits the output of register REG43 (fill element P) to register REG42, and multiplexer MUX43 transmits fill element P to register REG43. During the fourth period, multiplexer MUX40 transmits the output of register REG41 (element X03) to register REG40, multiplexer MUX41 transmits the output of register REG42 (fill element P) to register REG41, multiplexer MUX42 transmits the output of register REG43 (fill element P) to register REG42, and multiplexer MUX43 transmits fill element P to register REG43.

[0030] Figure 5 is a schematic diagram illustrating a convolution operation according to another application example. In the embodiment shown in Figure 5, it is assumed that the elements of matrix MX5 include X00, X01, X02, X03, X04, X05, X06, X07, X08, and X09, and the elements of convolution kernel MZ5 include Z00, Z01, Z02, and Z03. The stride parameter of the convolution operation is 2, and the padding parameter of the convolution operation is 0. The number of elements in matrix MX5 and convolution kernel MZ5 can be any real number determined according to the actual application. Referring to Figures 1 and 5, memory 210 is used to store matrix MX5 and convolution kernel MZ5. Based on the contents of memory 210, convolution operation circuit 220 can perform convolution operation to obtain operation result matrix MY5. The elements of operation result matrix MY5 include Y00, Y01, Y02, and Y03.

[0031] FIG6 is a circuit block diagram of a convolution operation device 600 according to another embodiment. The convolution operation device 600 shown in FIG6 includes memory 611, memory 612, memory 613, memory 614, memory 615 and convolution operation circuit 620. The memory 611 to 615 shown in FIG6 can be one of many embodiments of the memory 210 shown in FIG1. ​​Referring to FIG5 and FIG6, the elements Z00 to Z03 of the convolution kernel MZ5 are stored in memory 611. Elements X00, X04, and X08 (the first part) of matrix MX5 are stored in memory 612; elements X01, X05, and X09 (the second part) of matrix MX5 are stored in memory 613; elements X02 and X06 (the third part) of matrix MX5 are stored in memory 614; and elements X03 and X07 (the fourth part) of matrix MX5 are stored in memory 615. Different parts of matrix MX5 are stored in different memories 612-615 to avoid redundant storage of element data.

[0032] The convolution operation circuit 620 shown in FIG6 can be one of many implementation examples of the convolution operation circuit 220 shown in FIG1. ​​In the embodiment shown in FIG6, the convolution operation circuit 620 includes a routing and shift register circuit 621 and a plurality of multiply-accumulate circuits, such as multiply-accumulate circuits 622_0, 622_1, 622_2 and 622_3. Multiply-accumulate circuits 622_0 to 622_3 are coupled to memory 611 to access the convolution kernel MZ5. Multiply-accumulate circuit 622_0 corresponds to memory 612, multiply-accumulate circuit 622_1 corresponds to memory 613, multiply-accumulate circuit 622_2 corresponds to memory 614, and multiply-accumulate circuit 622_3 corresponds to memory 615. The multiply-accumulate circuits 622_0 to 622_3 and the routing and shift register circuit 621 shown in Figure 6 can be deduced by referring to the relevant descriptions of the multiply-accumulate circuits 422_0 to 422_3 and the routing and shift register circuit 421 shown in Figure 4, and will not be repeated here.

[0033] In the embodiment shown in FIG6, the routing and shift register circuit 621 includes multiplexer MUX60, register REG60, multiplexer MUX61, register REG62, multiplexer REG63, multiplexer MUX64, and multiplexer MUX65. This embodiment does not limit the specific implementation of multiplexers MUX60-MUX65 and registers REG60-REG63. For example, multiplexers MUX60-MUX65 can be conventional multiplexers or other data routing circuits, and registers REG60-REG63 can be conventional registers or other data temporary storage circuits.

[0034] One input terminal of multiplexer MUX60 is coupled to memory 612. The input terminal of register REG60 is coupled to the output terminal of multiplexer MUX60. The output terminal of register REG60 is coupled to multiply-accumulate circuit 622_0. One input terminal of multiplexer MUX61 is coupled to memory 613. The input terminal of register REG61 is coupled to the output terminal of multiplexer MUX61. The output terminal of register REG61 is coupled to another input terminal of multiplexer MUX60 and multiply-accumulate circuit 622_1. One input terminal of multiplexer MUX62 is coupled to memory 614. The input terminal of register REG62 is coupled to the output terminal of multiplexer MUX62. The output terminal of register REG62 is coupled to yet another input terminal of multiplexer MUX60, another input terminal of multiplexer MUX61, and multiply-accumulate circuit 622_2. One input of multiplexer MUX63 is coupled to memory 615. The input of register REG63 is coupled to the output of multiplexer MUX63. The output of register REG63 is coupled to another input of multiplexer MUX61, another input of multiplexer MUX62, and multiply-accumulate circuit 622_3. The other input of multiplexer MUX63 is coupled to the output of multiplexer MUX64.

[0035] The different inputs of the multiplexer MUX64 are coupled to memory 612, memory 613, memory 614, memory 615 and fill element P (e.g., 0 or other real number). The different inputs of the multiplexer MUX65 are coupled to memory 612, memory 613, memory 614, memory 615 and fill element P. The output of the multiplexer MUX65 is coupled to another input of the multiplexer MUX62.

[0036] During the first period, memory 611 provides element Z00 to multiply-accumulate circuits 622_0 to 622_3. Element X00 of memory 612 is transferred to multiply-accumulate circuit 622_0 via multiplexer MUX60 and register REG60. Element X01 of memory 613 is transferred to register REG61 via multiplexer MUX61. Element X02 of memory 614 is transferred to multiply-accumulate circuit 622_2 via multiplexer MUX62 and register REG62. Element X03 of memory 615 is transferred to register REG63 via multiplexer MUX63. Therefore, after the MAC operation in the first period, element Y00 is X00*Z00, and element Y01 is X02*Z00. During the first period, multiply-accumulate circuits 622_1 and 622_3 are idle, i.e., gated.

[0037] During the second period, memory 611 provides element Z02 to multiply-accumulate circuits 622_0 to 622_3. Element X02 of register REG62 is transferred to multiply-accumulate circuit 622_0 via multiplexer MUX60 and register REG60, and element X04 of memory 612 is transferred to multiply-accumulate circuit 622_2 via multiplexer MUX65, multiplexer MUX62 and register REG62. Therefore, after the MAC operation in the second period, element Y00 is X00*Z00 + X02*Z02, and element Y01 is X02*Z00 + X04*Z02. During the second period, multiply-accumulate circuits 622_1 and 622_3 are idle.

[0038] During the third period, memory 611 provides element Z01 to multiply-accumulate circuits 622_0 to 622_3. Element X01 of register REG61 is transferred to multiply-accumulate circuit 622_0 via multiplexer MUX60 and register REG60, and element X03 of register REG63 is transferred to multiply-accumulate circuit 622_2 via multiplexer MUX62 and register REG62. Therefore, after the MAC operation in the third period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02, and element Y01 is X02*Z00 + X03*Z01 + X04*Z02. During the third period, multiply-accumulate circuits 622_1 and 622_3 are idle.

[0039] During the fourth period, memory 611 provides element Z03 to multiply-accumulate circuits 622_0 to 622_3. Element X03 of register REG62 is transferred to multiply-accumulate circuit 622_0 via multiplexer MUX60 and register REG60. Element X05 of memory 613 is transferred to multiply-accumulate circuit 622_2 via multiplexer MUX65, multiplexer MUX62, and register REG62. Element X04 of memory 612 is transferred to register REG63 via multiplexer MUX64 and MUX63. Therefore, after the MAC operation in the fourth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X03*Z03, and element Y01 is X02*Z00 + X03*Z01 + X04*Z02 + X05*Z03. During the fourth period, the multiply-accumulate circuits 622_1 and 622_3 are idle.

[0040] During the fifth period, memory 611 provides element Z00 to multiply-accumulate circuits 622_0 to 622_3. Element X04 of register REG63 is transferred to multiply-accumulate circuit 622_1 via multiplexer MUX61 and register REG61. Element X06 of memory 614 is transferred to multiply-accumulate circuit 622_3 via multiplexer MUX64, multiplexer MUX63, and register REG63. Therefore, after the MAC operation in the fifth period, element Y02 is X04*Z00, and element Y03 is X06*Z00. During the fifth period, the multiply-accumulate circuits 622_0 and 622_2 are gated, so element Y00 remains X00*Z00 + X01*Z01 + X02*Z02 + X03*Z03, while element Y01 remains X02*Z00 + X03*Z01 + X04*Z02 + X05*Z03.

[0041] During the sixth period, memory 611 provides element Z02 to multiply-accumulate circuits 622_0 to 622_3. Element X06 of register REG63 is transferred to multiply-accumulate circuit 622_1 via multiplexer MUX61 and register REG61, and element X08 of memory 612 is transferred to multiply-accumulate circuit 622_3 via multiplexer MUX64, multiplexer MUX63 and register REG63. Therefore, after the MAC operation in the sixth period, element Y02 is X04*Z00 + X06*Z02, and element Y03 is X06*Z00 + X08*Z02. During the sixth period, multiply-accumulate circuits 622_0 and 622_2 are gated.

[0042] During the seventh period, memory 611 provides element Z01 to multiply-accumulate circuits 622_0 to 622_3. Element X05 of memory 613 is transferred to multiply-accumulate circuit 622_1 via multiplexer MUX61 and temporary register REG61, and element X07 of memory 615 is transferred to multiply-accumulate circuit 622_3 via multiplexer MUX63 and temporary register REG63. Therefore, after the MAC operation in the seventh period, element Y02 is X04*Z00 + X05*Z01 + X06*Z02, and element Y03 is X06*Z00 + X07*Z01 + X08*Z02. During the seventh period, multiply-accumulate circuits 622_0 and 622_2 are gated.

[0043] During the eighth period, memory 611 provides element Z03 to multiply-accumulate circuits 622_0 to 622_3. Element X07 of register REG63 is transferred to multiply-accumulate circuit 622_1 via multiplexer MUX61 and register REG61, and element X09 of memory 613 is transferred to multiply-accumulate circuit 622_3 via multiplexer MUX64, multiplexer MUX63 and register REG63. Therefore, after the MAC operation in the eighth period, element Y02 is X04*Z00 + X05*Z01 + X06*Z02 + X07*Z03, and element Y03 is X06*Z00 + X07*Z01 + X08*Z02 + X09*Z03. During the eighth period, multiply-accumulate circuits 622_0 and 622_2 are gated.

[0044] FIG7 is a circuit block diagram of a convolution operation device 700 according to another embodiment. The convolution operation device 700 shown in FIG7 includes memory 711, memory 712, memory 713, memory 714, memory 715 and convolution operation circuit 720. The memory 711 to 715 shown in FIG7 can be one of many embodiments of the memory 210 shown in FIG1. ​​Referring to FIG5 and FIG7, the elements Z00 to Z03 of the convolution kernel MZ5 are stored in memory 711. Elements X00, X01, X08, and X09 (the first part) of matrix MX5 are stored in memory 712; elements X02 and X03 (the second part) of matrix MX5 are stored in memory 713; elements X04 and X05 (the third part) of matrix MX5 are stored in memory 714; and elements X06 and X07 (the fourth part) of matrix MX5 are stored in memory 715. Different parts of matrix MX5 are stored in different memories 712 to 715 to avoid redundant storage of element data.

[0045] The convolution operation circuit 720 shown in FIG7 can be one of many implementation examples of the convolution operation circuit 220 shown in FIG1. ​​In the embodiment shown in FIG7, the convolution operation circuit 720 includes a routing and shift register circuit 721 and a plurality of multiply-accumulate circuits, such as multiply-accumulate circuits 722_0, 722_1, 722_2 and 722_3. Multiply-accumulate circuits 722_0 to 722_3 are coupled to memory 711 to access the convolution kernel MZ5. Multiply-accumulate circuit 722_0 corresponds to memory 712, multiply-accumulate circuit 722_1 corresponds to memory 713, multiply-accumulate circuit 722_2 corresponds to memory 714, and multiply-accumulate circuit 722_3 corresponds to memory 715. The multiply-accumulate circuits 722_0 to 722_3 and the routing and shift register circuit 721 shown in Figure 7 can be deduced by referring to the relevant descriptions of the multiply-accumulate circuits 422_0 to 422_3 and the routing and shift register circuit 421 shown in Figure 4, and will not be repeated here.

[0046] In the embodiment shown in FIG. 7, the routing and shift register circuit 721 includes multiplexer MUX70, register REG70, multiplexer MUX71, register REG72, multiplexer REG73, multiplexer REG74, and multiplexer MUX70-MUX74. This embodiment does not limit the specific implementation of multiplexers MUX70-MUX74 and registers REG70-REG73. For example, multiplexers MUX70-MUX74 can be conventional multiplexers or other data routing circuits, and registers REG70-REG73 can be conventional registers or other data temporary storage circuits.

[0047] One input terminal of multiplexer MUX70 is coupled to memory 712. The input terminal of register REG70 is coupled to the output terminal of multiplexer MUX70. The output terminal of register REG70 is coupled to multiply-accumulate circuit 722_0. One input terminal of multiplexer MUX71 is coupled to memory 713. The input terminal of register REG71 is coupled to the output terminal of multiplexer MUX71. The output terminal of register REG71 is coupled to the other input terminal of multiplexer MUX70 and multiply-accumulate circuit 722_1. One input terminal of multiplexer MUX72 is coupled to memory 714. The input terminal of register REG72 is coupled to the output terminal of multiplexer MUX72. The output terminal of register REG72 is coupled to the other input terminal of multiplexer MUX71 and multiply-accumulate circuit 722_2. One input terminal of multiplexer MUX73 is coupled to memory 715. The input of register REG73 is coupled to the output of multiplexer MUX73. The output of register REG73 is coupled to another input of multiplexer MUX72 and multiply-accumulate circuit 722_3. The other input of multiplexer MUX73 is coupled to the output of multiplexer MUX74. The different inputs of multiplexer MUX74 are coupled to memory 712, memory 713, memory 714, memory 715 and fill element P (e.g., 0 or other real number).

[0048] During the first period, memory 711 provides element Z00 to multiply-accumulate circuits 722_0 to 722_3, element X00 of memory 712 is transferred to multiply-accumulate circuit 722_0 through multiplexer MUX70 and register REG70, element X02 of memory 713 is transferred to multiply-accumulate circuit 722_1 through multiplexer MUX71 and register REG71, element X04 of memory 714 is transferred to multiply-accumulate circuit 722_2 through multiplexer MUX72 and register REG72, and element X06 of memory 715 is transferred to multiply-accumulate circuit 722_3 through multiplexer MUX73 and register REG73. Therefore, after the MAC operation in the first period, element Y00 is X00*Z00, element Y01 is X02*Z00, element Y02 is X04*Z00, and element Y03 is X06*Z00.

[0049] During the second period, memory 711 provides element Z02 to multiply-accumulate circuits 722_0 to 722_3. Element X02 of register REG71 is transferred to multiply-accumulate circuit 722_0 via multiplexer MUX70 and register REG70. Element X04 of register REG72 is transferred to multiply-accumulate circuit 722_1 via multiplexer MUX71 and register REG71. Element X06 of register REG73 is transferred to multiply-accumulate circuit 722_2 via multiplexer MUX72 and register REG72. Element X08 of memory 712 is transferred to multiply-accumulate circuit 722_3 via multiplexer MUX74, multiplexer MUX73 and register REG73. Therefore, after the MAC operation in the second period, element Y00 is X00*Z00 + X02*Z02, element Y01 is X02*Z00 + X04*Z02, element Y02 is X04*Z00 + X06*Z02, and element Y03 is X06*Z00 + X08*Z02.

[0050] During the third period, memory 711 provides element Z01 to multiply-accumulate circuits 722_0 to 722_3. Element X01 of memory 712 is transferred to multiply-accumulate circuit 722_0 via multiplexer MUX70 and register REG70. Element X03 of memory 713 is transferred to multiply-accumulate circuit 722_1 via multiplexer MUX71 and register REG71. Element X05 of memory 714 is transferred to multiply-accumulate circuit 722_2 via multiplexer MUX72 and register REG72. Element X07 of memory 715 is transferred to multiply-accumulate circuit 722_3 via multiplexer MUX73 and register REG73. Therefore, after the MAC operation in the third period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02, element Y01 is X02*Z00 + X03*Z01 + X04*Z02, element Y02 is X04*Z00 + X05*Z01 + X06*Z02, and element Y03 is X06*Z00 + X07*Z01 + X08*Z02.

[0051] During the fourth period, memory 711 provides element Z03 to multiply-accumulate circuits 722_0 to 722_3. Element X03 of register REG71 is transferred to multiply-accumulate circuit 722_0 via multiplexer MUX70 and register REG70. Element X05 of register REG72 is transferred to multiply-accumulate circuit 722_1 via multiplexer MUX71 and register REG71. Element X07 of register REG73 is transferred to multiply-accumulate circuit 722_2 via multiplexer MUX72 and register REG72. Element X09 of memory 712 is transferred to multiply-accumulate circuit 722_3 via multiplexer MUX74, multiplexer MUX73 and register REG73. Therefore, after the MAC operation in the fourth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X03*Z03, element Y01 is X02*Z00 + X03*Z01 + X04*Z02 + X05*Z03, element Y02 is X04*Z00 + X05*Z01 + X06*Z02 + X07*Z03, and element Y03 is X06*Z00 + X07*Z01 + X08*Z02 + X09*Z03.

[0052] Figure 8 is a schematic diagram illustrating a convolution operation according to another application example. In the embodiment shown in Figure 8, it is assumed that the elements of matrix MX8 include X00, X01, X02, X03, X04, X05, X10, X11, X12, X13, X14, X15, X20, X21, X22, X23, X24, and X25, and the elements of convolution kernel MZ8 include Z00, Z01, Z02, Z10, Z11, Z12, Z20, Z21, and Z22. The stride parameter of the convolution operation is 1, and the padding parameter of the convolution operation is 0. The number of elements in matrix MX8 and convolution kernel MZ8 can be any real number determined according to the actual application. Referring to Figures 1 and 8, memory 210 is used to store matrix MX8 and convolution kernel MZ8. Based on the contents of memory 210, convolution operation circuit 220 can perform convolution operation to obtain operation result matrix MY8. The elements of operation result matrix MY8 include Y00, Y01, Y02 and Y03.

[0053] FIG9 is a circuit block diagram of a convolution operation device 900 according to another embodiment. The convolution operation device 900 shown in FIG9 includes memory 911, memory 912, memory 913, memory 914, memory 915 and convolution operation circuit 920. The memory 911 to 915 shown in FIG9 can be one of many embodiments of the memory 210 shown in FIG1. ​​Referring to FIG8 and FIG9, the elements Z00 to Z22 of the convolution kernel MZ8 are stored in memory 911. Elements X00, X04, X10, X14, X20, and X24 (the first part) of matrix MX8 are stored in memory 912; elements X01, X05, X11, X15, X21, and X25 (the second part) of matrix MX8 are stored in memory 913; elements X02, X12, and X22 (the third part) of matrix MX8 are stored in memory 914; and elements X03, X13, and X23 (the fourth part) of matrix MX8 are stored in memory 915. Different parts of matrix MX8 are stored in different memories 912-915 to avoid redundant storage of element data.

[0054] The convolution operation circuit 920 shown in FIG9 can be one of many implementation examples of the convolution operation circuit 220 shown in FIG1. ​​In the embodiment shown in FIG9, the convolution operation circuit 920 includes a routing and shift register circuit 921 and multiple multiply-accumulate circuits, such as multiply-accumulate circuits 922_0, 922_1, 922_2 and 922_3. Multiply-accumulate circuits 922_0 to 922_3 are coupled to memory 911 to access the convolution kernel MZ8. Multiply-accumulate circuit 922_0 corresponds to memory 912, multiply-accumulate circuit 922_1 corresponds to memory 913, multiply-accumulate circuit 922_2 corresponds to memory 914, and multiply-accumulate circuit 922_3 corresponds to memory 915. The multiply-accumulate circuits 922_0 to 922_3 and the routing and shift register circuit 921 shown in Figure 9 can be deduced by referring to the relevant descriptions of the multiply-accumulate circuits 422_0 to 422_3 and the routing and shift register circuit 421 shown in Figure 4, and will not be repeated here.

[0055] In the embodiment shown in FIG9, the routing and shift register circuit 921 includes multiplexer MUX90, register REG90, multiplexer MUX91, register REG92, multiplexer MUX93, register REG94, and multiplexer MUX90-MUX94. This embodiment does not limit the specific implementation of multiplexers MUX90-MUX94 and registers REG90-REG93. For example, multiplexers MUX90-MUX94 can be conventional multiplexers or other data routing circuits, and registers REG90-REG93 can be conventional registers or other data temporary storage circuits.

[0056] One input terminal of multiplexer MUX90 is coupled to memory 912. The input terminal of register REG90 is coupled to the output terminal of multiplexer MUX90. The output terminal of register REG90 is coupled to multiply-accumulate circuit 922_0. One input terminal of multiplexer MUX91 is coupled to memory 913. The input terminal of register REG91 is coupled to the output terminal of multiplexer MUX91. The output terminal of register REG91 is coupled to the other input terminal of multiplexer MUX90 and multiply-accumulate circuit 922_1. One input terminal of multiplexer MUX92 is coupled to memory 914. The input terminal of register REG92 is coupled to the output terminal of multiplexer MUX92. The output terminal of register REG92 is coupled to the other input terminal of multiplexer MUX91 and multiply-accumulate circuit 922_2. One input terminal of multiplexer MUX93 is coupled to memory 915. The input of register REG93 is coupled to the output of multiplexer MUX93. The output of register REG93 is coupled to another input of multiplexer MUX92 and multiply-accumulate circuit 922_3. The other input of multiplexer MUX93 is coupled to the output of multiplexer MUX94. The different inputs of multiplexer MUX94 are coupled to memory 912, memory 913, memory 914, memory 915 and fill element P (e.g., 0 or other real number).

[0057] During the first period, memory 911 provides element Z00 to multiply-accumulate circuits 922_0 to 922_3, element X00 of memory 912 is transferred to multiply-accumulate circuit 922_0 through multiplexer MUX90 and register REG90, element X01 of memory 913 is transferred to multiply-accumulate circuit 922_1 through multiplexer MUX91 and register REG91, element X02 of memory 914 is transferred to multiply-accumulate circuit 922_2 through multiplexer MUX92 and register REG92, and element X03 of memory 915 is transferred to multiply-accumulate circuit 922_3 through multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the first period, element Y00 is X00*Z00, element Y01 is X01*Z00, element Y02 is X02*Z00, and element Y03 is X03*Z00.

[0058] During the second period, memory 911 provides element Z01 to multiply-accumulate circuits 922_0 to 922_3. Element X01 of register REG91 is transferred to multiply-accumulate circuit 922_0 via multiplexer MUX90 and register REG90. Element X02 of register REG92 is transferred to multiply-accumulate circuit 922_1 via multiplexer MUX91 and register REG91. Element X03 of register REG93 is transferred to multiply-accumulate circuit 922_2 via multiplexer MUX92 and register REG92. Element X04 of memory 912 is transferred to multiply-accumulate circuit 922_3 via multiplexer MUX94, multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the second period, element Y00 is X00*Z00 + X01*Z01, element Y01 is X01*Z00 + X02*Z01, element Y02 is X02*Z00 + X03*Z01, and element Y03 is X03*Z00 + X04*Z01.

[0059] During the third period, memory 911 provides element Z02 to multiply-accumulate circuits 922_0 to 922_3. Element X02 of register REG91 is transferred to multiply-accumulate circuit 922_0 via multiplexer MUX90 and register REG90. Element X03 of register REG92 is transferred to multiply-accumulate circuit 922_1 via multiplexer MUX91 and register REG91. Element X04 of register REG93 is transferred to multiply-accumulate circuit 922_2 via multiplexer MUX92 and register REG92. Element X05 of memory 913 is transferred to multiply-accumulate circuit 922_3 via multiplexer MUX94, multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the third period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02, element Y01 is X01*Z00 + X02*Z01 + X03*Z02, element Y02 is X02*Z00 + X03*Z01 + X04*Z02, and element Y03 is X03*Z00 + X04*Z01 + X05*Z02.

[0060] During the fourth period, memory 911 provides element Z10 to multiply-accumulate circuits 922_0 to 922_3. Element X10 of memory 912 is transferred to multiply-accumulate circuit 922_0 via multiplexer MUX90 and register REG90. Element X11 of memory 913 is transferred to multiply-accumulate circuit 922_1 via multiplexer MUX91 and register REG91. Element X12 of memory 914 is transferred to multiply-accumulate circuit 922_2 via multiplexer MUX92 and register REG92. Element X13 of memory 915 is transferred to multiply-accumulate circuit 922_3 via multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the fourth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X10*Z10, element Y01 is X01*Z00 + X02*Z01 + X03*Z02 + X11*Z10, element Y02 is X02*Z00 + X03*Z01 + X04*Z02 + X12*Z10, and element Y03 is X03*Z00 + X04*Z01 + X05*Z02 + X13*Z10.

[0061] During the fifth period, memory 911 provides element Z11 to multiply-accumulate circuits 922_0 to 922_3. Element X11 of register REG91 is transferred to multiply-accumulate circuit 922_0 via multiplexer MUX90 and register REG90. Element X12 of register REG92 is transferred to multiply-accumulate circuit 922_1 via multiplexer MUX91 and register REG91. Element X13 of register REG93 is transferred to multiply-accumulate circuit 922_2 via multiplexer MUX92 and register REG92. Element X14 of memory 912 is transferred to multiply-accumulate circuit 922_3 via multiplexer MUX94, multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the fifth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X10*Z10 + X11*Z11, element Y01 is X01*Z00 + X02*Z01 + X03*Z02 + X11*Z10 + X12*Z11, element Y02 is X02*Z00 + X03*Z01 + X04*Z02 + X12*Z10 + X13*Z11, and element Y03 is X03*Z00 + X04*Z01 + X05*Z02 + X13*Z10 + X14*Z11.

[0062] During the sixth period, memory 911 provides element Z12 to multiply-accumulate circuits 922_0 to 922_3. Element X12 of register REG91 is transferred to multiply-accumulate circuit 922_0 via multiplexer MUX90 and register REG90. Element X13 of register REG92 is transferred to multiply-accumulate circuit 922_1 via multiplexer MUX91 and register REG91. Element X14 of register REG93 is transferred to multiply-accumulate circuit 922_2 via multiplexer MUX92 and register REG92. Element X15 of memory 913 is transferred to multiply-accumulate circuit 922_3 via multiplexer MUX94, multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the sixth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X10*Z10 + X11*Z11 + X12*Z12, element Y01 is X01*Z00 + X02*Z01 + X03*Z02 + X11*Z10 + X12*Z11 + X13*Z12, element Y02 is X02*Z00 + X03*Z01 + X04*Z02 + X12*Z10 + X13*Z11 + X14*Z12, and element Y03 is X03*Z00 + X04*Z01 + X05*Z02 + X13*Z10 + X14*Z11 + X15*Z12.

[0063] During the seventh period, memory 911 provides element Z20 to multiply-accumulate circuits 922_0 to 922_3, element X20 of memory 912 is transferred to multiply-accumulate circuit 922_0 through multiplexer MUX90 and register REG90, element X21 of memory 913 is transferred to multiply-accumulate circuit 922_1 through multiplexer MUX91 and register REG91, element X22 of memory 914 is transferred to multiply-accumulate circuit 922_2 through multiplexer MUX92 and register REG92, and element X23 of memory 915 is transferred to multiply-accumulate circuit 922_3 through multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the seventh period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X10*Z10 + X11*Z11 + X12*Z12 + X20*Z20, element Y01 is X01*Z00 + X02*Z01 + X03*Z02 + X11*Z10 + X12*Z11 + X13*Z12 + X21*Z20, element Y02 is X02*Z00 + X03*Z01 + X04*Z02 + X12*Z10 + X13*Z11 + X14*Z12 + X22*Z20, and element Y03 is X03*Z00 + X04*Z01 + X05*Z02 + X13*Z10 + X14*Z11 + X15*Z12 + X23*Z20.

[0064] During the eighth period, memory 911 provides element Z21 to multiply-accumulate circuits 922_0 to 922_3. Element X21 of register REG91 is transferred to multiply-accumulate circuit 922_0 via multiplexer MUX90 and register REG90. Element X22 of register REG92 is transferred to multiply-accumulate circuit 922_1 via multiplexer MUX91 and register REG91. Element X23 of register REG93 is transferred to multiply-accumulate circuit 922_2 via multiplexer MUX92 and register REG92. Element X24 of memory 912 is transferred to multiply-accumulate circuit 922_3 via multiplexer MUX94, multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the eighth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X10*Z10 + X11*Z11 + X12*Z12 + X20*Z20 + X21*Z21, element Y01 is X01*Z00 + X02*Z01 + X03*Z02 + X11*Z10 + X12*Z11 + X13*Z12 + X21*Z20 + X22*Z21, element Y02 is X02*Z00 + X03*Z01 + X04*Z02 + X12*Z10 + X13*Z11 + X14*Z12 + X22*Z20 + X23*Z21, and element Y03 is X03*Z00 + X04*Z01 + X05*Z02 + X13*Z10 + X14*Z11 + X15*Z12 + X23*Z20 + X24*Z21.

[0065] During the ninth period, memory 911 provides element Z22 to multiply-accumulate circuits 922_0 to 922_3. Element X22 of register REG91 is transferred to multiply-accumulate circuit 922_0 via multiplexer MUX90 and register REG90. Element X23 of register REG92 is transferred to multiply-accumulate circuit 922_1 via multiplexer MUX91 and register REG91. Element X24 of register REG93 is transferred to multiply-accumulate circuit 922_2 via multiplexer MUX92 and register REG92. Element X25 of memory 913 is transferred to multiply-accumulate circuit 922_3 via multiplexer MUX94, multiplexer MUX93 and register REG93. Therefore, after the MAC operation in the ninth period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02 + X10*Z10 + X11*Z11 + X12*Z12 + X20*Z20 + X21*Z21 + X22*Z22, element Y01 is X01*Z00 + X02*Z01 + X03*Z02 + X11*Z10 + X12*Z11 + X13*Z12 + X21*Z20 + X22*Z21 + X23*Z22, and element Y02 is X02*Z00 + X03*Z01 + X04*Z02 + X12*Z10 + X13*Z11 + X14*Z12 + X22*Z20 + X23*Z21 + X24*Z22, and element Y03 is X03*Z00 + X04*Z01 + X05*Z02 + X13*Z10 + X14*Z11 + X15*Z12 + X23*Z20 + X24*Z21 + X25*Z22.

[0066] Figure 10 is a schematic diagram illustrating a convolution operation according to a further application example. In the embodiment shown in Figure 10, it is assumed that the elements of matrix MX10 include X00, X01, X02, X03, X04, and X05, the elements of convolution kernel MZ10 include Z00, Z01, and Z02, the stride parameter of the convolution operation is 1, and the padding parameter of the convolution operation is 0. The number of elements in matrix MX10 and convolution kernel MZ10 can be any real number determined according to the actual application. Referring to Figures 1 and 10, memory 210 is used to store matrix MX10 and convolution kernel MZ10. Based on the contents of memory 210, convolution operation circuit 220 can perform convolution operation to obtain operation result matrix MY10. The elements of operation result matrix MY10 include Y00, Y01, Y02, and Y03.

[0067] FIG11 is a circuit block diagram of a convolution operation device 1100 according to another embodiment. The convolution operation device 1100 shown in FIG11 includes memory 1111, memory 1112, memory 1113, memory 1114, memory 1115 and convolution operation circuit 1120. The memory 1111 to 1115 shown in FIG11 can be one of many embodiments of the memory 210 shown in FIG1. ​​Referring to FIG10 and FIG11, the elements Z00 to Z02 of the convolution kernel MZ10 are stored in memory 1111. Elements X00 and X04 (the first part) of matrix MX10 are stored in memory 1112, elements X01 and X05 (the second part) of matrix MX10 are stored in memory 1113, element X02 (the third part) of matrix MX10 is stored in memory 1114, and element X03 (the fourth part) of matrix MX10 is stored in memory 1115. Different parts of matrix MX10 are stored in different memories 1112 to 1115 to avoid redundant storage of element data.

[0068] The convolution operation circuit 1120 shown in FIG11 can be one of many implementation examples of the convolution operation circuit 220 shown in FIG1. ​​In the embodiment shown in FIG11, the convolution operation circuit 1120 includes a routing and shift register circuit 1121 and a plurality of multiply-accumulate circuits, such as multiply-accumulate circuits 1122_0, 1122_1, 1122_2 and 1122_3. Multiply-accumulate circuits 1122_0 to 1122_3 are coupled to memory 1111 to access the convolution kernel MZ10. Multiply-accumulate circuit 1122_0 corresponds to memory 1112, multiply-accumulate circuit 1122_1 corresponds to memory 1113, multiply-accumulate circuit 1122_2 corresponds to memory 1114, and multiply-accumulate circuit 1122_3 corresponds to memory 1115. The multiply-accumulate circuits 1122_0 to 1122_3 and the routing and shift register circuit 1121 shown in Figure 11 can be deduced by referring to the relevant descriptions of the multiply-accumulate circuits 422_0 to 422_3 and the routing and shift register circuit 421 shown in Figure 4, and will not be repeated here.

[0069] In the embodiment shown in FIG11, the routing and shift register circuit 1121 includes multiplexer MUX110, register REG110, multiplexer MUX111, register REG111, multiplexer MUX112, register REG112, multiplexer MUX113, and multiplexer MUX114. This embodiment does not limit the specific implementation of multiplexers MUX110-MUX114 and registers REG110-REG112. For example, multiplexers MUX110-MUX114 can be conventional multiplexers or other data routing circuits, and registers REG110-REG112 can be conventional registers or other data temporary storage circuits.

[0070] One input of multiplexer MUX110 is coupled to memory 1112. The output of multiplexer MUX110 is coupled to multiply-accumulate circuit 1122_0. One input of multiplexer MUX111 is coupled to memory 1113. The output of multiplexer MUX111 is coupled to multiply-accumulate circuit 1122_1. The input of register REG110 is coupled to the output of multiplexer MUX111. The output of register REG110 is coupled to another input of multiplexer MUX110. One input of multiplexer MUX112 is coupled to memory 1114. The output of multiplexer MUX112 is coupled to multiply-accumulate circuit 1122_2. The input of register REG111 is coupled to the output of multiplexer MUX112. The output of register REG111 is coupled to another input of multiplexer MUX111. One input of multiplexer MUX113 is coupled to memory 1115. The output of multiplexer MUX113 is coupled to multiply-accumulate circuit 1122_3. The input of register REG112 is coupled to the output of multiplexer MUX113. The output of register REG112 is coupled to another input of multiplexer MUX112. Different inputs of multiplexer MUX114 are coupled to memory 1112, memory 1113, memory 1114, memory 1115, and a fill element P (e.g., 0 or another real number). The output of multiplexer MUX114 is coupled to another input of multiplexer MUX113.

[0071] During the first period, memory 1111 provides element Z00 to multiply-accumulate circuits 1122_0 to 1122_3, element X00 of memory 1112 is transferred to multiply-accumulate circuit 1122_0 through multiplexer MUX110, element X01 of memory 1113 is transferred to multiply-accumulate circuit 1122_1 and temporary register REG110 through multiplexer MUX111, element X02 of memory 1114 is transferred to multiply-accumulate circuit 1122_2 and temporary register REG111 through multiplexer MUX112, and element X03 of memory 1115 is transferred to multiply-accumulate circuit 1122_3 and temporary register REG112 through multiplexer MUX113. Therefore, after the MAC operation in the first period, element Y00 is X00*Z00, element Y01 is X01*Z00, element Y02 is X02*Z00, and element Y03 is X03*Z00.

[0072] During the second period, memory 1111 provides element Z01 to multiply-accumulate circuits 1122_0 to 1122_3. Element X01 of register REG110 is transferred to multiply-accumulate circuit 1122_0 via multiplexer MUX110. Element X02 of register REG111 is transferred to multiply-accumulate circuit 1122_1 and register REG110 via multiplexer MUX111. Element X03 of register REG112 is transferred to multiply-accumulate circuit 1122_2 and register REG111 via multiplexer MUX112. Element X04 of memory 1112 is transferred to multiply-accumulate circuit 1122_3 and register REG112 via multiplexers MUX114 and MUX113. Therefore, after the MAC operation in the second period, element Y00 is X00*Z00 + X01*Z01, element Y01 is X01*Z00 + X02*Z01, element Y02 is X02*Z00 + X03*Z01, and element Y03 is X03*Z00 + X04*Z01.

[0073] During the third period, memory 1111 provides element Z02 to multiply-accumulate circuits 1122_0 to 1122_3. Element X02 of register REG110 is transferred to multiply-accumulate circuit 1122_0 via multiplexer MUX110. Element X03 of register REG111 is transferred to multiply-accumulate circuit 1122_1 and register REG110 via multiplexer MUX111. Element X04 of register REG112 is transferred to multiply-accumulate circuit 1122_2 and register REG111 via multiplexer MUX112. Element X05 of memory 1113 is transferred to multiply-accumulate circuit 1122_3 and register REG112 via multiplexers MUX114 and MUX113. Therefore, after the MAC operation in the third period, element Y00 is X00*Z00 + X01*Z01 + X02*Z02, element Y01 is X01*Z00 + X02*Z01 + X03*Z02, element Y02 is X02*Z00 + X03*Z01 + X04*Z02, and element Y03 is X03*Z00 + X04*Z01 + X05*Z02.

[0074] Although the present invention has been disclosed above by way of embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims. [Simplified Explanation of the Diagram]

[0007] FIG1 is a circuit block diagram of a convolution operation device according to an embodiment of the present invention. FIG2 is a schematic diagram of a convolution operation according to an application example. FIG3 is a circuit block diagram of a convolution operation device according to an embodiment. FIG4 is a circuit block diagram of a convolution operation device according to an embodiment of the present invention. FIG5 is a schematic diagram of a convolution operation according to another application example. FIG6 is a circuit block diagram of a convolution operation device according to another embodiment. FIG7 is a circuit block diagram of a convolution operation device according to yet another embodiment. FIG8 is a schematic diagram of a convolution operation according to another application example. FIG9 is a circuit block diagram of a convolution operation device according to yet another embodiment. FIG10 is a schematic diagram of a convolution operation according to a further application example. FIG11 is a circuit block diagram of a convolution operation device according to yet another embodiment.

Claims

1. A convolution operation device, comprising: The first memory is used to store a convolution kernel; A second memory is used to store a first element of a matrix; A third memory location is used to store a second element of the matrix; A first multiply-accumulate circuit is coupled to the first memory to access the convolution kernel; A second multiply-accumulate circuit is coupled to the first memory to access the convolution kernel; The system includes a routing and shift register circuit coupled to the second memory, the third memory, the first multiply-accumulate circuit, and the second multiply-accumulate circuit. During a first period, the routing and shift register circuit transfers the first element of the second memory to the first multiply-accumulate circuit, and the routing and shift register circuit transfers the second element of the third memory to the second multiply-accumulate circuit; and during a second period, the routing and shift register circuit transfers the second element of the third memory to the first multiply-accumulate circuit. The routing and shift register circuit includes: a first multiplexer having a first input coupled to the second memory; and a first register having an input coupled to an output of the first multiplexer, wherein an output of the first register is coupled to the first multiply-accumulate circuit. A second multiplexer having a first input coupled to the third memory; and a second register having an input coupled to an output of the second multiplexer, wherein an output of the second register is coupled to a second input of the first multiplexer and the second multiply-accumulate circuit.

2. The convolution operation apparatus as claimed in claim 1, wherein a first portion of the matrix is ​​stored in the second memory, a second portion of the matrix is ​​stored in the third memory, and the first portion is mutually exclusive with the second portion.

3. The convolution operation apparatus as claimed in claim 1, wherein the first multiply-accumulate circuit comprises: A temporary register having an input terminal coupled to the first memory; And a multiply-accumulate-adder having a first input coupled to an output of the register, wherein a second input of the multiply-accumulate-adder is coupled to the routing and shift register circuit.

4. The convolution operation apparatus as described in claim 1, further comprising: A fourth memory location is used to store a third element of the matrix; And a third multiply-accumulate circuit coupled to the first memory to access the convolution kernel, wherein, during the first period, the routing and shift register circuit further transfers the third element of the fourth memory to the third multiply-accumulate circuit; during the second period, the routing and shift register circuit further transfers the third element of the fourth memory to the second multiply-accumulate circuit; and during a third period, the routing and shift register circuit transfers the third element of the fourth memory to the first multiply-accumulate circuit.

5. The convolution operation apparatus as claimed in claim 1, wherein the routing and shift register circuitry further comprises: A third multiplexer having a first input terminal coupled to a fourth memory of the convolution operation device; And a third register having an input terminal coupled to an output terminal of the third multiplexer, wherein an output terminal of the third register is coupled to a second input terminal of the second multiplexer and a third multiply-accumulate circuit of the convolution operation device.

6. The convolution operation apparatus as claimed in claim 5, wherein a second input of the third multiplexer receives a padding element.

7. The convolution operation apparatus as claimed in claim 5, wherein an output of the third register is further coupled to a third input of the first multiplexer.

8. The convolution operation apparatus as claimed in claim 5, wherein the routing and shift register circuitry further comprises: A fourth multiplexer, wherein a first input terminal of the fourth multiplexer is coupled to the second memory, a second input terminal of the fourth multiplexer is coupled to the third memory, a third input terminal of the fourth multiplexer is coupled to the fourth memory, and an output terminal of the fourth multiplexer is coupled to a second input terminal of the third multiplexer.

9. The convolution operation apparatus as claimed in claim 8, wherein a fourth input of the fourth multiplexer receives a padding element.

10. The convolution operation apparatus as claimed in claim 8, wherein the routing and shift register circuitry further comprises: A fifth multiplexer, wherein a first input of the fifth multiplexer is coupled to the second memory, a second input of the fifth multiplexer is coupled to the third memory, a third input of the fifth multiplexer is coupled to the fourth memory, a fourth input of the fifth multiplexer receives a fill element, and an output of the fifth multiplexer is coupled to a third input of the second multiplexer.

11. A convolution operation device, comprising: The first memory is used to store a convolution kernel; A second memory is used to store a first element of a matrix; A third memory location is used to store a second element of the matrix; A first multiply-accumulate circuit is coupled to the first memory to access the convolution kernel; A second multiply-accumulate circuit is coupled to the first memory to access the convolution kernel; The system includes a routing and shift register circuit coupled to the second memory, the third memory, the first multiply-accumulate circuit, and the second multiply-accumulate circuit. During a first period, the routing and shift register circuit transfers the first element of the second memory to the first multiply-accumulate circuit, and the routing and shift register circuit transfers the second element of the third memory to the second multiply-accumulate circuit. During a second period, the routing and shift register circuit transfers the second element of the third memory to the first multiply-accumulate circuit. The routing and shift register circuit includes a first multiplexer having a first input coupled to the second memory, and an output coupled to the first multiply-accumulate circuit. A second multiplexer having a first input coupled to the third memory, wherein an output of the second multiplexer is coupled to the second multiply-accumulate circuit; and a first register having an input coupled to the output of the second multiplexer, wherein an output of the first register is coupled to a second input of the first multiplexer.

12. The convolution operation apparatus as claimed in claim 11, wherein the routing and shift register circuitry further comprises: A third multiplexer has a first input coupled to a fourth memory of the convolution operation device, wherein an output of the third multiplexer is coupled to a third multiply-accumulate circuit of the convolution operation device; And a second register having an input terminal coupled to an output terminal of the third multiplexer, wherein an output terminal of the second register is coupled to a second input terminal of the second multiplexer.

13. The convolution operation apparatus as claimed in claim 12, wherein the routing and shift register circuitry further comprises: A fourth multiplexer, wherein a first input terminal of the fourth multiplexer is coupled to the second memory, a second input terminal of the fourth multiplexer is coupled to the third memory, a third input terminal of the fourth multiplexer is coupled to the fourth memory, and an output terminal of the fourth multiplexer is coupled to a second input terminal of the third multiplexer.

14. The convolution operation apparatus as claimed in claim 13, wherein a fourth input of the fourth multiplexer receives a padding element.

Images