A configurable crossbar circuit and convolution operation circuit based thereon

Abstract

The application provides a configurable cross switch circuit and a convolution operation circuit based on the same, the configurable cross switch circuit comprising a read cross switch array in front of an operation unit array and a write cross switch array in front of the operation unit array; input feature maps are introduced into an input memory stack, input into the operation unit array through the read cross switch array, and output feature maps calculated by the operation unit array are written into an output memory stack through the write cross switch; in each cycle, at most only one switch is closed for the same column read switch and the same row write switch. The two-dimensional convolution operation circuit based on the application only needs to introduce the input feature map data into the on-chip memory once, and the two-dimensional convolution operation can be completed, without the need of repeatedly introducing the input feature map data into the memory or transmitting the input feature map data between the memories, so that the performance of the two-dimensional convolution calculation is improved and the energy consumption is reduced.

Description

Technical Field

[0001] This invention relates to the field of convolution operation technology, and in particular to a configurable cross switch circuit and a convolution operation circuit based thereon. Background Technology

[0002] Convolutional layers are one of the fundamental layers of convolutional neural networks. Convolution is one of the most common operations in deep learning, accounting for over 90% of the computational cost of convolutional neural networks. In recent years, with the widespread application of deep learning technology in fields such as computer vision and speech processing, many hardware accelerators have been proposed to improve the execution efficiency of deep learning models. Among them, systolic arrays are a common circuit that supports convolution operations. Figure 1 This diagram illustrates the computation of a 2D convolution operation using a systolic array with a 3x3 sliding window. The systolic array efficiently computes the output feature map of a single row of the 2D convolution operation. To compute the first row of output feature maps, the first row of input feature map data needs to be imported into the first memory block; the second row into the second memory block; and the third row into the third memory block. Similarly, to compute the second row of output feature maps, the second row of input feature map data needs to be imported into the first memory block; the third row into the second memory block; and the fourth row into the third memory block. Finally, to compute the third row of output feature maps, the third row of input feature map data needs to be imported into the first memory block; the fourth row into the second memory block; and the fifth row into the third memory block.

[0003] In the above example, during the entire convolution operation, the third row of input feature map data is repeatedly imported into three different memory locations within the memory heap. However, in integrated circuits, the energy consumed in data transmission is far greater than the energy consumed in data computation. Clearly, the above data import method will consume more energy, and repeated data transmission will inevitably lead to a decrease in the overall performance of the convolution computation. Summary of the Invention

[0004] To address the problem that existing systolic arrays require repeated data transmission for convolution operations, this invention proposes a configurable cross-switch circuit and a convolution operation circuit based thereon, which can flexibly import data stored in different memories into the corresponding rows of the systolic array.

[0005] This invention protects a configurable cross switch circuit, including a read cross switch array and a write cross switch array in front of the arithmetic unit array; the input feature map is imported into the input memory stack, and then input into the arithmetic unit array through the read cross switch array; the arithmetic unit array calculates the output feature map and writes it into the output memory stack through the write cross switch.

[0006] The read cross switch array consists of m rows and n columns of configurable read switches, where the number of rows m ≥ the number of input memories and the number of columns n ≥ the number of rows of arithmetic units. In each cycle, at most one read switch in the same column is closed.

[0007] The write crossbar switch array consists of p rows and q columns of configurable write switches, where the number of rows p ≥ the number of output memories and the number of columns q ≥ the number of arithmetic units. At most one write switch in each row is closed in each cycle.

[0008] Preferably, both the read switch and the write switch correspond to a 1-bit register and are configured via software.

[0009] The present invention also protects a convolution operation circuit based on the above-mentioned configurable cross switch circuit, which is composed of x rows and y columns of operation units, wherein the operation units are mainly composed of multipliers and accumulators.

[0010] The multiplier's row-direction input side inputs the input feature map corresponding to the row data of the convolutional layer. The convolution kernel weights are stored inside the operation unit before the convolution operation. The accumulator's part is the sum of the part of the output of the operation unit of the previous row input side.

[0011] By performing two-dimensional convolution operations using the configurable cross-switch circuit disclosed in this invention, the input feature map data only needs to be imported into the on-chip memory once to complete the two-dimensional convolution operation. This eliminates the need to repeatedly import the input feature map data into the memory or transfer the input feature map data between memory locations, thereby improving the performance of two-dimensional convolution calculations and reducing energy consumption. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating the data transmission method for the first row of the output feature map.

[0013] Figure 2 This is a block diagram of a convolution operation circuit based on a configurable cross switch circuit;

[0014] Figure 3 A schematic diagram of a cross switch;

[0015] Figure 4 Write a schematic diagram of a cross switch;

[0016] Figure 5 A schematic diagram of a 6×6 input feature map and a 3×3 convolution kernel;

[0017] Figure 6 This is a schematic diagram of the read crossbar switch status in the first row of the output feature map;

[0018] Figure 7 A schematic diagram of the write cross switch state for calculating the output feature map;

[0019] Figure 8 This is a schematic diagram illustrating the data transmission method for the second row of the output feature map.

[0020] Figure 9 This is a schematic diagram of the read crossbar switch status in the second row of the output feature map;

[0021] Figure 10 This is a schematic diagram illustrating the data transmission method for the third row of the output feature map.

[0022] Figure 11 This is a schematic diagram of the crossbar switch status in the third row of the output feature map;

[0023] Figure 12 This is a schematic diagram of the data transmission method for calculating the fourth row of the output feature map. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and design various embodiments with various modifications suitable for a particular purpose.

[0025] Example 1

[0026] A configurable cross switch circuit, such as Figure 2 As shown, it includes a read crossbar switch array on the front side of the arithmetic unit array and a write crossbar switch array on the front side of the arithmetic unit array; the input feature map is imported into the input memory stack, and then input into the arithmetic unit array through the read crossbar switch array; the arithmetic unit array calculates the output feature map and writes it into the output memory stack through the write crossbar switch.

[0027] The read cross switch array consists of m rows and n columns of configurable read switches. Figure 3 The diagram shows a 4-row, 8-column crossbar switch. The number of rows m ≥ the number of input memories, and the number of columns n ≥ the number of rows of arithmetic units. At most one switch in the same column is closed during each cycle.

[0028] The write crossbar switch array consists of p rows and q columns of configurable write switches. Figure 4 The diagram shows a 4-row, 8-column write crossbar switch. The number of rows p ≥ the number of output memories, and the number of columns q ≥ the number of arithmetic unit columns. At most one write switch in each row is closed per cycle.

[0029] The operation of the configurable cross switch circuit disclosed in this invention will be elaborated below with reference to a specific convolution operation circuit.

[0030] The convolution operation circuit consists of x rows and y columns of operation units, which mainly consist of multipliers and accumulators. The row input side of the multiplier receives the row data corresponding to the input feature map of the convolutional layer. The convolution kernel weights are stored inside the operation unit before the convolution operation. The accumulator's input side receives a partial sum of the output of the previous row of operation units. The aforementioned configurable cross-switch circuit is provided between the operation unit array and the input memory stack and output memory stack.

[0031] by Figure 5 As shown, taking a 6×6 matrix as the input feature map and a 3×3 matrix as the convolution kernel as an example, when calculating the convolution operation result of the kernel sliding across the first three rows of the input feature map matrix (i.e., calculating the first row of the output feature map), the data is processed according to... Figure 1 The input processing unit PE array shown is used to broadcast the data of the first row of the first input feature map to the first to third rows of the PE array, the data of the second row of the first input feature map to the fourth to sixth rows of the PE array, and the data of the third row of the first input feature map to the seventh to ninth rows of the PE array.

[0032] If there are multi-channel inputs ( Figure 1 If the input is a 2-channel input, then the weights for the convolutional window of the first input channel are arranged in rows 1-9, the weights for the convolutional window of the second input channel are arranged in rows 10-18, and so on. The input feature maps for different input channels are then input into the PE array along the corresponding rows of the weights for the convolutional windows of those input channels. For ease of explanation, the following description focuses on the first input feature map.

[0033] When calculating the output feature map of the convolution kernel in the second row of the input feature map matrix, the data of the second row of the first input feature map is broadcast into rows 1-3 of the PE array, the data of the third row of the first input feature map is broadcast into rows 4-6 of the PE array, and the data of the fourth row of the first input feature map is broadcast into rows 7-9 of the PE array; when calculating the output feature map of the convolution kernel in the third row of the input feature map matrix, the data of the third row of the first input feature map is broadcast into rows 1-3 of the PE array, the data of the fourth row of the first input feature map is broadcast into rows 4-6 of the PE array, and the data of the fifth row of the first input feature map is broadcast into rows 7-9 of the PE array; and so on.

[0034] Traditionally, the memory is directly and fixedly connected to the computation unit. This results in some row input feature map data being repeatedly imported into different memories within the memory stack, reducing the overall performance of convolution computation, consuming more energy, and wasting resources. The configurable cross-switch circuit proposed in this invention, where each read and write switch corresponds to a 1-bit register, can be configured via software to flexibly import data from the memory stack into different locations within the computation unit.

[0035] When calculating the output feature map of the convolution kernel in the first row of the input feature map matrix, the data transmission method is as follows: Figure 1 As shown, the first row of data from the input feature map stored in input memory 0 is broadcast to rows 1-3 of the PE array; the second row of data from the input feature map stored in input memory 1 is broadcast to rows 4-6 of the PE array; and the third row of data from the input feature map stored in input memory 2 is broadcast to rows 7-9 of the PE array. The read crossbar switch circuit is configured as follows: Figure 6 As shown. Due to Figure 1 The shown arithmetic unit has only 4 columns, so output memory 0-3 is sufficient. The write crossbar switch circuit is configured as follows: Figure 7 As shown.

[0036] When calculating the output feature map of the second row of the input feature map matrix using this convolutional kernel, the data transmission method is as follows: Figure 8 As shown. At this point, the second and third rows of the input feature map remain in their original positions. The first row of the input feature map is no longer needed after the first row of the output feature map is calculated and can be overwritten by new data. Therefore, the fourth row of the input feature map is imported into input memory 0. At this time, memory 0, memory 1, and memory 2 store the fourth, second, and third rows of the input feature map, respectively. The second row of the output feature map is calculated using a two-dimensional convolution operation. The data must be processed according to... Figure 8 The transmission is performed as shown. At this time, the configuration status of the cross switch is as follows: Figure 9 As shown, the configuration status of the cross switch remains as follows. Figure 7 As shown.

[0037] When the third row of the output feature map is calculated, the data transmission method is as follows: Figure 10 As shown. At this point, the data in rows 4 and 3 of the input feature map remain in their original positions, while the data in row 2 of the input feature map is no longer needed after the calculation of row 2 of the output feature map and can be overwritten by new data. Therefore, the data in row 5 of the input feature map is imported into input memory 1. At this time, memory 0, memory 1, and memory 2 store the data in rows 4, 5, and 3 of the input feature map, respectively. The data in row 3 of the output feature map is calculated using a two-dimensional convolution operation, and the data must be processed according to... Figure 9 The transmission is performed as shown. At this time, the configuration status of the cross switch is as follows: Figure 11 As shown, the configuration status of the cross switch remains as follows. Figure 7 As shown.

[0038] When the fourth row of the output feature map is calculated, the data transmission method is as follows: Figure 12As shown. At this point, the data in rows 4 and 5 of the input feature map remain in their original positions, while the data in row 3 of the input feature map is no longer needed after the calculation of row 3 of the output feature map and can be overwritten by new data. The data in row 6 of the input feature map is then imported into input memory 2. At this time, memory 0, memory 1, and memory 2 store the data in rows 4, 5, and 6 of the input feature map, respectively. The data in row 4 of the output feature map is calculated using a two-dimensional convolution operation, and the data must be processed according to... Figure 12 The transmission is performed as shown. At this time, the configuration state of the cross switch is restored to [normal state]. Figure 6 As shown, the configuration status of the cross switch remains as follows. Figure 7 As shown.

[0039] When calculating and outputting other rows of the feature map, the same approach is followed to ensure that the input feature map data remains in memory as much as possible, avoiding data movement between on-chip and off-chip memory, or between on-chip memory locations. By configuring the "read crossbar switch" and "write crossbar switch" with different data transfer modes, data in memory can be flexibly transferred to the computation unit, allowing two-dimensional convolution operations to be performed according to specific rules even if the data remains stationary in a certain memory location.

[0040] Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art and related fields based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

Claims

1. A configurable cross switch circuit, characterized in that, It includes a read crossbar switch array on the front side of the arithmetic unit array and a write crossbar switch array on the back side of the arithmetic unit array; the input feature map is imported into the input memory stack, and then input into the arithmetic unit array through the read crossbar switch array; the arithmetic unit array calculates the output feature map and writes it into the output memory stack through the write crossbar switch. The read cross switch array consists of m rows and n columns of configurable read switches, where the number of rows m ≥ the number of input memories and the number of columns n ≥ the number of rows of arithmetic units. In each cycle, at most one read switch in the same column is closed. The write cross switch array consists of p rows and q columns of configurable write switches, where the number of rows p ≥ the number of output memories and the number of columns q ≥ the number of arithmetic units. In each cycle, at most one write switch in each row is closed. Both the read switch and the write switch correspond to a 1-bit register, which can be configured via software.

2. A convolution operation circuit, characterized in that, It consists of x rows and y columns of operation units, and the operation units are mainly composed of multipliers and accumulators; The multiplier inputs the row feature map corresponding to the row data of the convolutional layer on the row direction input side. The convolutional kernel weights are stored inside the operation unit before the convolution operation begins. The accumulator's part is the sum of the part output of the previous row operation unit input on the input side. The configurable cross switch circuit as described in claim 1 is provided between the arithmetic unit array and the input memory stack and the output memory stack.

Images