AI accelerator and electronic equipment

Abstract

The invention discloses an AI accelerator and electronic equipment. The AI accelerator comprises a control unit, a data processing unit and an operation unit, wherein the control unit is suitable for controlling the work of the data conversion unit and the operation unit based on the operation indication information; the data processing unit is suitable for acquiring to-be-processed data under the control of the control unit and processing the to-be-processed data to obtain first operation data; the operation unit is suitable for executing the operation identified by the operation identification information on the first operation data under the control of the control unit; wherein the operation indication information comprises operation identification information and operation parameter information, and the operation identification information is suitable for identifying the currently executed operation as convolution operation or fast Fourier transform operation. By adopting the scheme, the processing duration of the FFT task can be shortened, and the resource utilization rate of the AI accelerator can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an AI accelerator and electronic equipment. Background Art

[0002] Artificial Intelligence (AI) chips, also known as AI accelerators, are modules specifically designed to handle large amounts of computing tasks in AI applications and are widely used in satellites, remote sensing, autonomous driving and other fields.

[0003] When using AI accelerators to process artificial intelligence tasks, it is necessary to process (Fast Fourier Transform, FFT) tasks in upstream and downstream tasks. Current AI accelerators do not directly support FFT calculations, resulting in the need for external computing devices to implement such tasks.

[0004] When an external computing device is used to process FFT tasks, the operating frequency of the external computing device is much lower than that of the AI ​​accelerator, which results in a longer time required for the external computing device to process the FFT task. In addition, when the external computing device is processing the FFT task, the AI ​​accelerator is usually idle, resulting in low resource utilization of the AI ​​accelerator. Summary of the invention

[0005] The problem to be solved by the present invention is: how to shorten the processing time of the FFT task and improve the resource utilization of the AI ​​accelerator.

[0006] To solve the above problems, an embodiment of the present invention provides an AI accelerator, which includes: a control unit, a data processing unit, and a computing unit; wherein:

[0007] The control unit is adapted to receive operation instruction information and control the operation of the data conversion unit and the operation unit based on the operation instruction information;

[0008] The data processing unit is adapted to obtain the data to be processed under the control of the control unit, process the data to be processed, obtain the first operation data, and output the first operation data to the first input terminal of the operation unit;

[0009] The operation unit is adapted to perform the operation identified by the operation identification information on the first operation data under the control of the control unit;

[0010] The operation indication information includes operation identification information and operation parameter information, wherein the operation identification information is suitable for identifying that the operation currently being performed is a convolution operation or a fast Fourier transform operation, and the operation parameter information is suitable for providing the operation parameters required for the operation currently being performed.

[0011] Optionally, the control unit is further adapted to obtain other operation data corresponding to the operation indication information, and output the other operation data to the second input terminal of the operation unit.

[0012] Optionally, the computing unit includes:

[0013] A Fourier transform operation subunit, adapted to perform a Fourier transform operation on the first operation data using the operation data inputted from the second input terminal;

[0014] and a complex multiplication subunit adapted to perform a complex multiplication operation on the operation result of the Fourier transform operation subunit.

[0015] Optionally, the Fourier transform operator unit includes a plurality of multiplication and addition modules, and the plurality of multiplication and addition modules are systolic connected to form a systolic array; an output end of the systolic array is connected to an input end of the complex multiplication subunit.

[0016] Optionally, the multiplication-addition module comprises: a first selection submodule and a multiplication-addition submodule; wherein:

[0017] The first selection submodule is adapted to select a first multiplier of the multiplication-addition submodule; the first multiplier is the other operation data, or is output data of the previous multiplication-addition module;

[0018] The multiplication-addition submodule is adapted to perform a multiplication-addition operation on the first multiplier selected by the first selection submodule.

[0019] Optionally, the complex multiplication subunit comprises: a plurality of complex multipliers; the input ends of the complex multipliers are connected to the output ends of two columns of multiplication-addition modules in the systolic array that perform Fourier transform operations on the same data.

[0020] Optionally, the complex multiplier comprises: a second selection submodule and a multiplication submodule; wherein:

[0021] The second selection submodule is adapted to select the second multiplier and the third multiplier of the multiplication submodule; the second multiplier and the third multiplier are preset fixed values, or are output data of the previous multiplication and addition module;

[0022] The multiplication submodule is adapted to perform a multiplication operation on the second multiplier selected by the second selection submodule.

[0023] Optionally, the operation identified by the operation identification information is a fast Fourier transform operation, and the operation parameter information includes: sampling point number information and cardinality information.

[0024] Optionally, the data to be processed is a time domain signal; the data processing unit is suitable for sampling the time domain signal conversion according to the sampling point number information to obtain each sampling value, and converting each sampling value into a corresponding complex matrix according to the cardinality information, and using the complex matrix corresponding to the sampling value as the first operation data.

[0025] Optionally, two adjacent row elements in the same column of the complex matrix are real data and imaginary data of the same sampling value, or two adjacent column elements in the same row of the complex matrix are real data and imaginary data of the same sampling value.

[0026] Optionally, the other operation data are Fourier transform operation coefficients corresponding to the operation parameter information, or are rotation factors.

[0027] Optionally, the control unit is adapted to pre-store the Fourier transform operation coefficients in the Fourier transform operator unit, and determine a process of a fast Fourier transform operation based on the sampling point number information and the cardinality information, and adjust the inputs of the first input terminal and the second input terminal of the operation unit according to the determined process of the fast Fourier transform operation;

[0028] The process of the fast Fourier transform operation includes the number of times the Fourier transform operation is performed, the number of times the rotation factor is updated, and the order of performing the Fourier transform operation and the complex multiplication operation.

[0029] Optionally, the operation identified by the operation identification information is a convolution operation, and the operation parameter information includes: size information of a convolution kernel.

[0030] Optionally, the data to be processed is input feature data; and the data processing unit is suitable for converting the input feature data into a three-dimensional matrix corresponding to the operation unit.

[0031] Optionally, the other operation data is a one-dimensional convolution kernel obtained after the convolution kernel is split.

[0032] Optionally, the operation unit further includes: at least one delay subunit.

[0033] An embodiment of the present invention further provides an electronic device, comprising any one of the above-mentioned AI accelerators. Compared with the prior art, the technical solution of the embodiment of the present invention has the following advantages:

[0034] By applying the solution of the present invention, a control unit, a data processing unit and an operation unit are provided in the AI ​​accelerator. The data processing unit can process the data to be processed to form the first operation data. The operation unit can perform a convolution operation or a fast Fourier transform operation on the first operation data under the control of the control unit. Thus, the AI ​​accelerator can complete both the convolution operation and the fast Fourier transform operation. In this way, there is no need to use an external computing device to perform the fast Fourier transform operation. By using the high computing speed of the AI ​​accelerator itself, the duration of the fast Fourier transform operation can be effectively shortened, and the duration of the fast Fourier transform operation can be further shortened by reducing the data transmission time. In addition, the resource utilization of the AI ​​accelerator is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the processing process of an FFT task;

[0036] Figure 2 is a schematic diagram of the structure of an AI accelerator in an embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of the structure of another AI accelerator in an embodiment of the present invention;

[0038] Figure 4 is a structural diagram of a multiplication-addition module in an embodiment of the present invention;

[0039] Figure 5 is a structural schematic diagram of a complex multiplier in an embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of a fast Fourier transform process in an embodiment of the present invention;

[0041] Figure 7 is a schematic diagram of the principle of a fast Fourier transform in an embodiment of the present invention;

[0042] Figure 8 1 is a schematic diagram of a convolution operation process in an embodiment of the present invention;

[0043] Fig. 9 Schematic diagram of a convolution kernel decomposition in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] Figure 1 The following is a schematic diagram of the processing process of the existing FFT task. Figure 1Since the current AI accelerator 11 does not support FFT calculations, in order to complete the FFT task, a common practice is to take the data to be processed from the AI ​​accelerator 11 to the external computing device 12, and the external computing device 12 performs FFT calculations on the taken data. After the calculation is completed, the calculation result is returned to the AI ​​accelerator 11 for subsequent calculations.

[0045] The above solution has the following disadvantages when completing the FFT task:

[0046] 1) Due to the complexity of FFT calculation, the external computing device 12 needs to use the FFT intellectual property (IP) core of the field programmable gate array (FPGA) for calculation. However, the operating frequency of the FPGA's FFT IP core is in the order of 100Mhz, which is much lower than the operating frequency of the AI ​​accelerator 11 designed by the application specific integrated circuit (ASIC). The operating frequency of the AI ​​accelerator 11 is in the order of 1Ghz, which makes the calculation speed of the FPGA's FFT IP core lower than the calculation speed of the AI ​​accelerator 11. As a result, the FFT calculation, which originally accounted for a small proportion of the AI ​​task, occupies a large amount of computing time.

[0047] 2) Since the AI ​​accelerator and the external computing device 12 do not directly share storage, additional bandwidth is required for data transmission, and the bandwidth also takes up a lot of time.

[0048] 3) When the data to be processed is calculated in the external computing device 12, the AI ​​accelerator 11 is in an idle state and the resource utilization rate is not high.

[0049] In view of this problem, the present invention provides an AI accelerator, which can not only complete the convolution operation, but also complete the fast Fourier transform operation. In this way, there is no need to use an external computing device to perform the fast Fourier transform operation. The high computing speed of the AI ​​accelerator itself can effectively shorten the duration of the fast Fourier transform operation, and the duration of the fast Fourier transform operation can be further shortened by reducing the data transmission time. In addition, the resource utilization of the AI ​​accelerator is significantly improved.

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0051] Reference Figure 2, an embodiment of the present invention provides an AI accelerator, the AI ​​accelerator comprising: a control unit 21, a data processing unit 22 and a computing unit 23. Among them:

[0052] The control unit 21 is adapted to receive operation instruction information and control the operation of the data conversion unit 22 and the operation unit 23 based on the operation instruction information;

[0053] The data processing unit 22 is adapted to obtain the data to be processed under the control of the control unit 21, and process the data to be processed to obtain the first operation data, and output the first operation data to the first input terminal of the operation unit 23;

[0054] The operation unit 23 is adapted to perform the operation identified by the operation identification information on the first operation data under the control of the control unit 21;

[0055] The operation indication information includes operation identification information and operation parameter information, wherein the operation identification information is suitable for identifying that the operation currently being performed is a convolution operation or a fast Fourier transform operation, and the operation parameter information is suitable for providing the operation parameters required for the operation currently being performed.

[0056] Under the control of the control unit 21, the computing unit 23 can not only complete the convolution operation, but also complete the fast Fourier transform operation without the aid of an external computing device. Thus, relying on the high computing speed of the AI ​​accelerator, the duration of the fast Fourier transform operation can be effectively shortened, and the resource utilization of the AI ​​accelerator can be improved.

[0057] In a specific implementation, the control unit 21 is further adapted to obtain other operation data corresponding to the operation indication information, and output the other operation data to the second input terminal of the operation unit 23 .

[0058] Specifically, when the operation operation identified by the operation identification information is a fast Fourier transform operation operation, the other operation data may be a Fourier transform operation coefficient corresponding to the operation parameter information, or a rotation factor, and the specific time of inputting the Fourier transform operation coefficient and the rotation factor may be set according to the process of the fast Fourier transform operation operation. When the operation operation identified by the operation identification information is a convolution operation operation, the other operation data may be a 1-dimensional convolution kernel obtained by splitting the convolution kernel of the convolution operation operation, and the 1-dimensional convolution kernel has the same number of input channels as the convolution kernel of the convolution operation operation.

[0059] In a specific implementation, the computing unit 23 may have a variety of implementation modes, which are not limited here.

[0060] In one embodiment, referring to Figure 2 , the operation unit 23 may include: a Fourier transform operation subunit 231 and a complex multiplication subunit 232. Wherein:

[0061] The Fourier transform operator 231 is adapted to perform a Fourier transform operation on the first operation data using the operation data inputted from the second input terminal;

[0062] The complex multiplication subunit 232 is adapted to perform a complex multiplication operation on the operation result of the Fourier transform operation subunit.

[0063] Specifically, by changing the operation data input to the Fourier transform operation subunit 231 and the complex multiplication subunit 232, a fast Fourier transform operation and a convolution operation can be implemented.

[0064] In one embodiment, referring to Figure 3 The Fourier transform operator unit 231 may include a plurality of multiplication-addition modules PE, and the plurality of multiplication-addition modules PE are pulsatingly connected to form a pulsating array; the output end of the pulsating array is connected to the input end of the complex multiplication subunit 232.

[0065] In one embodiment, referring to Figure 3 The complex multiplication subunit 232 may include: a plurality of complex multipliers C_Mul; the input end of the complex multiplier C_Mul is connected to the output end of two columns of multiplication-addition modules in the systolic array that perform Fourier transform operations on the same data.

[0066] Specifically, in the multiplication-addition modules PE in the same row of the systolic array, an output data of the previous multiplication-addition module PE is used as the input of the next multiplication-addition module PE. In the multiplication-addition modules PE in the same column, an output data of the previous multiplication-addition module PE is used as the input of the next multiplication-addition module PE. In the multiplication-addition modules PE in the same column, the output end of the bottom multiplication-addition module PE is the output end of the multiplication-addition module PE in the column. For the convenience of description, the previous multiplication-addition module PE of the current multiplication-addition module PE is called the previous-stage multiplication-addition module PE of the current multiplication-addition module PE. The previous multiplication-addition module PE of the current multiplication-addition module PE is called the previous-stage multiplication-addition module PE of the current multiplication-addition module PE.

[0067] Typically, the output ends of two adjacent columns of multiplication-addition modules PE in the systolic array correspond to the real part and the imaginary part of the same operation data input by the control unit 21, respectively. Therefore, the output ends of two adjacent columns of multiplication-addition modules PE in the systolic array are connected to the input ends of the same complex multiplier C_Mul, so as to perform a complex multiplication operation on the output data of the multiplication-addition module PE and the input data of the control unit 21.

[0068] It should be noted that Figure 3The systolic array shown in the figure is composed of 8 rows and 8 columns of multiplication-addition modules PE, but it is only an embodiment. In practical applications, the size of the systolic array can be set according to actual computing requirements. For example, the systolic array can also be composed of 16 rows and 8 columns of multiplication-addition modules PE, or 4 rows and 4 columns of multiplication-addition modules PE.

[0069] In one embodiment, referring to Figure 4 , the multiplication-addition module PE may include: a first selection submodule 41 and a multiplication-addition submodule 42. Wherein:

[0070] The first selection submodule 41 is adapted to select a first multiplier of the multiplication-addition submodule 42; the first multiplier is the other operation data, or is output data of the previous multiplication-addition module;

[0071] The multiplication-addition submodule 42 is adapted to perform a multiplication-addition operation on the first multiplier selected by the first selection submodule 41 .

[0072] The multiplication-addition module PE is a controllable multiplication-addition module, and specifically can control the first multiplier of the multiplication-addition module PE based on the first control signal s1 output by the control unit 21. Specifically, the multiplication-addition module PE can receive the first input data Input_a from the previous stage multiplication-addition module, and receive the second input data Input_b and the third input data Input_c of the previous stage multiplication-addition module. The multiplication-addition module PE can also include a first register R1, and a second register R2 and a third register R3 connected to the multiplication-addition submodule 42.

[0073] in:

[0074] The first input data Input_a can be directly used as the first output data Output_a of the multiplication-addition module PE.

[0075] The first register R1 may store Fourier transform operation coefficients in advance. The first selection submodule 41 may select whether to import the second input data Input_b of the previous multiplication and addition module into the first register R1 and participate in the calculation according to the first control signal s1. When the first selection submodule 41 imports the second input data Input_b of the previous multiplication and addition module into the first register R1, the first multiplier selected by the first selection submodule 41 is the second input data Input_b of the previous multiplication and addition module. When the first selection submodule 41 does not import the second input data Input_b of the previous multiplication and addition module into the first register R1, the first multiplier selected by the first selection submodule 41 is the Fourier transform operation coefficient.

[0076] The second register R2 can store the first multiplier selected by the first selection submodule 41. When the first multiplier is needed to participate in the subsequent operation, the first multiplier can be read from the second register R2 and used as the second output data Output_b of the multiplication and addition module.

[0077] In a specific implementation, the multiplication-addition submodule 42 may include a first multiplier M1 and a first adder A1. The first multiplier M1 may perform a multiplication operation on the first multiplier selected by the first selection submodule 41 and the first input data Input_a of the previous multiplication-addition module. The first adder A1 may perform a summation operation on the multiplication operation result of the multiplier M1 and the third input data Input_c, and the summation operation result may be stored in a third register R3. When the summation operation result is required to participate in the subsequent operation, the summation operation result may be used as the third output data Output_c of the multiplication-addition module PE. The third output data Output_c is a partial sum, and the partial sum is transmitted and accumulated in the systolic array until the accumulation is completed at the bottom of the systolic array.

[0078] In a specific implementation, the complex multiplier C_Mul can perform complex calculation on the second output data Output_b and the third output data Output_c of the two multiplication and addition modules PE of the previous stage under the control of the control unit. Figure 5 , the second output data of one of the multiplication-addition modules PE can be recorded as Output_b1, and the third output data can be recorded as Output_c1, and the second output data of the other multiplication-addition module PE can be recorded as Output_b2, and the third output data can be recorded as Output_c2.

[0079] In one embodiment, referring to Figure 5 , the complex multiplier C_Mul may include: a second selection submodule 51 and a multiplication submodule 52. Wherein:

[0080] The second selection submodule 51 is adapted to select the second multiplier and the third multiplier of the multiplication submodule 52; the second multiplier and the third multiplier are preset fixed values, or are output data of the previous multiplication and addition module;

[0081] The multiplication submodule 52 is adapted to perform a multiplication operation on the second multiplier selected by the second selection submodule 51 .

[0082] In a specific implementation, the preset fixed value may be 1 or 0. Specifically, the second selection submodule 51 may include two selectors, namely a first selector 511 and a second selector 512. The first selector 511 may select the second output data Output_b1 of the two multiplication-addition modules PE of the previous stage or the value 1 as the second multiplier of the multiplication submodule 52 under the control of the second control signal s2 output by the control unit. The second selector 512 may select the second output data Output_b2 of the two multiplication-addition modules PE of the previous stage or the value 0 as the third multiplier of the multiplication submodule 52 under the control of the second control signal s2 output by the control unit.

[0083] When performing a convolution operation, the first selector 511 can select a value of 1 as the second multiplier of the multiplication submodule 52, and the second selector 512 can select a value of 0 as the third multiplier of the multiplication submodule 52. When performing a fast Fourier transform operation, the first selector 511 can select the second output data Output_b1 as the second multiplier of the multiplication submodule 52. The second selector 512 can select the second output data Output_b2 as the third multiplier of the multiplication submodule 52.

[0084] When a value of 0 or 1 is selected as the multiplier of the multiplication submodule 52, the output result of the operation unit can be not affected by the complex multiplier C_Mul, that is, the output result of the operation unit remains the output result of the Fourier transform operation subunit.

[0085] In a specific implementation, the multiplication submodule 52 may adopt a three-multiplier structure. Specifically, the multiplication submodule 52 may include: a second multiplier M2, a third multiplier M3, and a fourth multiplier M4. In addition to the multipliers, the multiplication submodule 52 may also include: a second adder A2, a third adder A3, a fourth adder A4, a fifth adder A5, and a sixth adder A6. Wherein:

[0086] The second adder A2 can sum the output of the first selector 511 and the output of the second selector 512, and after the second multiplier M2 multiplies the summed result with the third output data Output_c1 of the previous multiplication-addition module, the summed result is input to the sixth adder A6. The fourth adder can sum the third output data Output_c2 of the previous multiplication-addition module and the third output data Output_c1 of the previous multiplication-addition module, and after the third multiplier M3 multiplies the summed result and the output of the second selector 512, the multiplication result is output to the fifth adder A5 and the sixth adder A6.

[0087] The sixth adder A6 can sum the output of the third multiplier M3 and the output of the second multiplier M2, and the sum result is used as the first output data Output_c1' of the complex multiplier C_Mul. The fourth multiplier M4 can multiply the output of the fourth adder A4 and the output of the third adder A3, and output the multiplication result to the fourth multiplier M4. The fifth adder A5 sums the output of the fourth multiplier M4 and the output of the third multiplier M3, and the sum result is used as the second output data Output_c2' of the complex multiplier C_Mul.

[0088] At this time, the first output data Output_c1 ′ and the second output data Output_c2 ′ of the complex multiplier C_Mul are respectively the real part data and the imaginary part data of an operation result in the fast Fourier transform operation.

[0089] During the fast Fourier transform operation, the complex multiplication operation performed by the multiplication submodule 52 can be expressed as:

[0090] (Output b1 +i*Output b2 )*(Output c1 +i*Output c2 )=(Output b1 *Output c1 -Output b2 Output c2 )+i*(Output b1 *Output c2 +

[0091] Output b2 *Output c1 )=Output c1 ′+i*Output c2 '

[0092] In a specific implementation, in order to align the input data of each multiplication-addition module PE and the complex multiplier C_Mul, the operation unit may further include at least one delay subunit. Figure 3 The operation unit may include: a first delay subunit 233, which may include a plurality of delays, such as delays z-1 to z-7. By setting the first delay subunit 233, the input data of each row can be aligned with the output data of the previous multiplication and addition module PE.

[0093] In some embodiments, the operation unit may further include: a second delay subunit 233. The second delay subunit 233 may also include a plurality of delays, such as delays z-1 to z-7. By setting the second delay subunit 233, the output data of the previous multiplication and addition module PE can be aligned with the input data of each row.

[0094] The following is a detailed description of the process of the AI ​​accelerator performing the fast Fourier transform operation and the convolution operation:

[0095] When the operation operation identified by the operation identification information is a fast Fourier transform operation operation, the operation parameter information includes: sampling point number information and cardinality information. At this time, the data to be processed is a time domain signal, and the data processing unit can sample the time domain signal conversion according to the sampling point number information to obtain each sampling value, convert each sampling value into a corresponding complex matrix according to the cardinality information, and use the complex matrix corresponding to the sampling value as the first operation data.

[0096] For example, when the number of sampling points is 16 and the base is 4, the fast Fourier transform operation is a 16-point base 4 fast Fourier transform operation. Specifically, the time domain signal x(t) is sampled, and the sampling values ​​obtained are: x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, x11, x12, x13, x14, x15. According to the base 4, the sampling values ​​are divided into 4 columns, and the obtained sampling matrix can be expressed as:

[0097]

[0098] In a specific implementation, after obtaining the above-mentioned sampling matrix, the above-mentioned sampling matrix can also be converted into a corresponding complex matrix to perform a complex multiplication operation, wherein the complex matrix corresponding to the above-mentioned 16-point base 4 sampling matrix, that is, each sampling value in the sampling matrix is ​​represented in complex form. Specifically, when the operation unit calculates w*x (where x represents the sampling matrix, and w represents the rotation factor or DFT coefficient), the elements of two adjacent rows in the same column of the complex matrix can be the real data and imaginary data of the same sampling value. When the operation unit calculates x*w, the elements of two adjacent columns in the same row of the complex matrix can be the real data and imaginary data of the same sampling value.

[0099] Take the calculation of x*w by the operation unit as an example. At this time, x is used as the row input data of the systolic array, and w is used as the column input data of the systolic array. At this time, the complex matrix corresponding to the above sampling matrix is ​​as follows Figure 7The data in the middle area 71 are shown. Among them, x0.r represents the real data of the sample value x0, and x0.i represents the imaginary data of the sample value x0. Similarly, x14.0 represents the real data of the sample value x14, and x0.i represents the imaginary data of the sample value x14.

[0100] In the embodiment of the present invention, in order to realize the complex number calculation in the fast Fourier transform operation, the real part and the imaginary part of the same data need to be calculated separately. Therefore, when two complex numbers are multiplied, if the real part and the imaginary part are multiplied separately, a 2*2 systolic array is required. There is no difference between complex number accumulation and real number accumulation, so when the Fourier transform operation is performed on the above 4*4 sampling matrix, an 8*8 systolic array is required.

[0101] In the specific implementation, taking the above sampling matrix as an example, refer to Figure 6 , the complex form corresponding to the first column of the sampling matrix (i.e., x0, x4, x8, and x12) can be first input into the systolic array to perform a Fourier transform operation, and then the complex form corresponding to the second column of the sampling matrix (i.e., x1, x5, x9, and x13) can be input into the systolic array to perform a Fourier transform operation, and then the complex form corresponding to the third column of the sampling matrix (i.e., x2, x6, x10, and x14) can be input into the systolic array to perform a Fourier transform operation, and finally the complex form corresponding to the third column of the sampling matrix (i.e., x3, x7, x11, and x15) can be input into the systolic array to perform a Fourier transform operation.

[0102] In a specific implementation, the control unit is suitable for pre-storing the Fourier transform operation coefficients into the Fourier transform operator unit, and based on the sampling point number information and cardinality information, determining the process of the fast Fourier transform operation operation, and adjusting the input of the first input terminal and the second input terminal of the operation unit according to the determined process of the fast Fourier transform operation operation. The process of the fast Fourier transform operation operation includes the number of times the Fourier transform operation operation is performed, the number of times the rotation factor is updated, and the order of performing the Fourier transform operation operation and the complex multiplication operation. By determining the process of the fast Fourier transform operation operation, the Fourier transform operation operation process can be decomposed into stages, so that the number of parameters and the number of scheduling times can be improved through the systolic array method to improve the computing efficiency.

[0103] In a specific implementation, assuming that the number of sampling points is N and the cardinality is M, in one fast Fourier transform operation, logM(N) Fourier transform operations can be performed and [logM(N)-1] rotation factors can be updated.

[0104] For example, refer to Figure 6For the above-mentioned 16-point radix 4 sampling matrix, each fast Fourier transform operation requires log4(16)=2 Fourier transform operations and updates the rotation factor once. Specifically, when executing the fast Fourier transform operation result, for the first column of the above-mentioned 16-point radix 4 sampling matrix, its complex form can be first input into the systolic array to perform a 4-point Fourier transform operation, and then the Fourier transform operation and the rotation factor are multiplied, and finally the Fourier transform operation and the rotation factor are multiplied and then a 4-point Fourier transform operation is performed to obtain the final fast Fourier transform operation result of the first column. Afterwards, the above process is repeated for the second column of the above-mentioned 16-point radix 4 sampling matrix until each column obtains the fast Fourier transform operation result.

[0105] For another example, when the number of sampling points is 64, each fast Fourier transform operation requires log4(64)=3 Fourier transform operations and updates the rotation factor twice.

[0106] It should be noted that, in the process of performing a fast Fourier transform operation on the same column of the sampling matrix, the matrix Y output by a 4-point Fourier transform operation and a rotation factor multiplication operation needs to be transposed before the next 4-point Fourier transform operation. At this time, the control unit can obtain the matrix Y and input it into the systolic array in a transposed form.

[0107] It should be noted that the radix and number of sampling points of the fast Fourier transform operation should match the size of the systolic array. For example, if the fast Fourier transform operation is performed using the radix-4 method, the systolic array can be an 8*8 array. If the fast Fourier transform operation is performed using the radix-2 method, the systolic array can be a 4*4 array. If the fast Fourier transform operation is performed using the radix-8 method, the systolic array can be a 16*16 array.

[0108] In a specific implementation, the control unit may obtain other operation data required for the fast Fourier transform operation, wherein the other operation data may be Fourier transform operation coefficients corresponding to the operation parameter information, or rotation factors.

[0109] The Fourier transform coefficients are determined only by the number of points. The coefficients of all 4-point DFTs are the same, and the coefficients of all 8-point DFTs are also the same. The Fourier transform coefficients can be pre-stored in the first register R1 of the multiplication and addition module PE (e.g. Figure 4 When performing the Fourier transform operation, the stored Fourier transform operation coefficients are taken out from the first register R1, and subsequent multiplication and addition operations are performed.

[0110] When performing a complex multiplication operation, the control unit can input the rotation factor as the second input data Input_b to the complex multiplier. For fast Fourier transform operations with different sampling points, the value of the rotation factor is different, and it is possible that the rotation factors in each complex multiplier are different at the same time.

[0111] Taking the above 16-point base 4 sampling matrix as an example, the corresponding rotation factor matrix can be expressed as:

[0112]

[0113] in, When N=16, Convert the above sampling rotation factor matrix into the corresponding complex matrix, such as Figure 7 As shown in the data in the middle area 72. In the rotation factor complex matrix, the elements of two adjacent rows in the same column can be the real data and imaginary data of the same sample value. For example, the real data corresponding to the rotation factor w0 is w0.r, and the imaginary data is w0.i.

[0114] When the fast Fourier transform operation is performed on the first column in the sampling matrix, the corresponding rotation factor is the first row of the rotation factor matrix (i.e., w0, w0, w0). When the fast Fourier transform operation is performed on the second column in the sampling matrix, the corresponding rotation factor is the second row of the rotation factor matrix (i.e., w0, w1, w2, w3). When the fast Fourier transform operation is performed on the third column in the sampling matrix, the corresponding rotation factor is the third row of the rotation factor matrix (i.e., w0, w2, w4, w6). When the fast Fourier transform operation is performed on the fourth column in the sampling matrix, the corresponding rotation factor is the fourth row of the rotation factor matrix (i.e., w0, w3, w6, w9).

[0115] Reference Figure 4 and 7 Taking the above sampling matrix as an example, when calculating the Fourier transform operation, the multiplication and addition module PE first converts the Fourier transform operation coefficients (such as Figure 7 The data in the middle area 73 is stored in the first register R1 as the second input data Input_b and then remains unchanged.

[0116] During the formal calculation, the first column of the sampling matrix is ​​used as the first input data Input_a in the input value pulsation array, multiplied by the Fourier transform operation coefficient in the multiplication and addition module PE, and the partial sum Output_c is generated through the first adder A1 and passed downward until the last multiplication and addition module completes the accumulation calculation.

[0117] When calculating the multiplication of the rotation factors, the multiplication-addition module PE uses the rotation factors as the second input data Input_b, which does not participate in the calculation of the multiplication-addition module PE (at this time, the first control signal s1 always keeps the value of the first register R1 in the multiplication-addition module PE unit unchanged), and the rotation factors only participate in the calculation of the complex multiplier C_Mul and are always updated. After obtaining the multiplication result, it is output to the local storage.

[0118] When the rotation factor calculation is completed, the next stage of calculation can be to read the multiplication result from the local storage into the systolic array and calculate the next Fourier transform operation. The coefficients of the Fourier transform operation do not need to be changed, that is, the first register R1 continues to keep the original data unchanged.

[0119] The above process is performed alternately to complete the fast Fourier transform operation through the systolic array.

[0120] By controlling the pulsating array to multiply the rotation factors together immediately after each 4-point Fourier transform operation, it is not a simple Fourier transform operation. Instead, through stage decomposition, the butterfly unit is replaced by a Fourier transform operator unit, and the rotation factor is calculated in combination with a complex multiplier to implement a fast Fourier transform operation. This can reduce the number of parameters and the number of scheduling times, and effectively improve computing efficiency.

[0121] In addition, during the entire calculation process, only the decomposed Fourier transform operation coefficients (such as 4-point DFT, 8-point DFT) and the rotation factors of each stage need to be stored, which is consistent with the storage amount of coefficients in the fast Fourier transform operation.

[0122] In practical applications, the size of the systolic array and the specific scheduling method are set according to the requirements of the fast Fourier transform operation. If a 4096-point base 8 fast Fourier transform operation is used, the size of the MAC array is 16*16, the number of multipliers C_Mul can be 8, and there are a total of log8(4096)=4 stages. Except for the last stage, each stage has an 8-point DFT and a rotation factor multiplication, and a total of 4 8-point DFTs and 3 rotation factor multiplications are required.

[0123] In a specific implementation, when the operation operation identified by the operation identification information is a convolution operation operation, the operation parameter information includes: size information of the convolution kernel. At this time, the data to be processed may be input feature data. The data processing unit is adapted to convert the input feature data into three-dimensional matrix data corresponding to the operation unit.

[0124] For example, refer to Figure 8, the input feature data 81 may be an input feature map, where IF_C represents the number of input channels, IF_X represents the height of the input feature map, and IF_Y represents the width of the input feature map. The size of the convolution kernel 82 is (K_x, K_y, IF_C), where K_x represents the height of the convolution kernel, K_y represents the width of the convolution kernel, and OF_C represents the number of output channels.

[0125] When using the systolic array of the present invention to calculate convolution, the control unit needs to decompose the convolution kernel. A convolution kernel of size (K_x, K_y, IF_C) can be decomposed into K_x*K_y (1, 1, IF_C) convolution kernel units. When performing the convolution operation, the convolution operation of these convolution kernel units is calculated first, and then the entire large convolution kernel is traversed, such as Fig. 9 As shown. The convolution operation of each small convolution kernel unit is a one-dimensional (1D) convolution. By splitting the convolution kernel, the two-dimensional (2D) convolution operation is converted into a one-dimensional convolution operation. Among them, when IF_C is greater than the number of columns of the systolic array, IF_C can be differentiated, and the convolution operation when IF_C=8 is calculated first, and then the convolution operation when the remaining IF_C>8 is calculated. Each convolution kernel unit is used as the input data of a column of the systolic array, and the row input data of the systolic array is convolved, and the convolution operation result 83 is finally obtained (refer to Figure 8 ).

[0126] In a specific implementation, the input feature data is converted into a three-dimensional matrix corresponding to the operation unit, and the size of the three-dimensional matrix is ​​jointly defined by the number of rows, the number of columns and the number of input channels. The number of rows and the number of columns of the three-dimensional matrix are consistent with the number of rows and the number of columns of the systolic array, and the number of channels input to the systolic array each time needs to be set according to the size of the systolic array.

[0127] It should be noted that the systolic array in the embodiment of the present invention can not only realize two-dimensional convolution operations, but also realize convolution operations in more dimensions. Using a systolic array to complete a convolution operation belongs to the prior art. The solution of the present invention mainly utilizes the existing systolic array that can complete the convolution operation to realize the fast Fourier transform operation under the scheduling of the control unit.

[0128] From the above content, it can be seen that the AI ​​accelerator in the embodiment of the present invention can complete the convolution operation and the fast Fourier transform operation in the same chip without the help of external equipment, which can improve resource utilization and save task time.

[0129] Regarding the various modules / units included in the various devices and products described in the above embodiments, they can be software modules / units, or hardware modules / units, or they can be partially software modules / units and partially hardware modules / units. For example, for various devices and products applied to or integrated in a chip, the various modules / units included therein can all be implemented in the form of hardware such as circuits, or at least some of the modules / units can be implemented in the form of software programs, which run on a processor integrated inside the chip, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in a chip module, the various modules / units included therein can all be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (such as a chip, circuit module, etc.) or different components of the chip module, or at least some of the modules / units can be implemented in the form of hardware such as circuits. The element can be implemented in the form of a software program, which runs on a processor integrated inside the chip module, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits; for various devices and products applied to or integrated in the terminal, the various modules / units contained therein can be implemented in the form of hardware such as circuits, and different modules / units can be located in the same component (for example, chip, circuit module, etc.) or in different components in the terminal, or, at least some modules / units can be implemented in the form of a software program, which runs on a processor integrated inside the terminal, and the remaining (if any) modules / units can be implemented in the form of hardware such as circuits.

[0130] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. An AI accelerator, characterized in that: include: A control unit, a data processing unit and an operation unit; wherein: The control unit is adapted to receive operation instruction information and control the operation of the data conversion unit and the operation unit based on the operation instruction information; The data processing unit is adapted to obtain the data to be processed under the control of the control unit, process the data to be processed, obtain the first operation data, and output the first operation data to the first input terminal of the operation unit; The operation unit is adapted to perform the operation identified by the operation identification information on the first operation data under the control of the control unit; The operation indication information includes operation identification information and operation parameter information, wherein the operation identification information is suitable for identifying that the operation currently being performed is a convolution operation or a fast Fourier transform operation, and the operation parameter information is suitable for providing the operation parameters required for the operation currently being performed.

2. The AI ​​accelerator according to claim 1, wherein: The control unit is further adapted to obtain other operation data corresponding to the operation indication information and output the other operation data to the second input terminal of the operation unit.

3. The AI ​​accelerator according to claim 2, wherein: The computing unit comprises: A Fourier transform operation subunit, adapted to perform a Fourier transform operation on the first operation data using the operation data inputted from the second input terminal; and a complex multiplication subunit adapted to perform a complex multiplication operation on the operation result of the Fourier transform operation subunit.

4. The AI ​​accelerator according to claim 3, wherein: The Fourier transform operator unit includes a plurality of multiplication and addition modules, and the plurality of multiplication and addition modules are systolic connected to form a systolic array; the output end of the systolic array is connected to the input end of the complex multiplication subunit.

5. The AI ​​accelerator according to claim 4, wherein: The multiplication-addition module comprises: a first selection submodule and a multiplication-addition submodule; wherein: The first selection submodule is adapted to select a first multiplier of the multiplication-addition submodule; the first multiplier is the other operation data, or is output data of the previous multiplication-addition module; The multiplication-addition submodule is adapted to perform a multiplication-addition operation on the first multiplier selected by the first selection submodule.

6. The AI ​​accelerator according to claim 4, wherein: The complex multiplication subunit comprises: a plurality of complex multipliers; the input end of the complex multiplier is connected to the output end of two columns of multiplication-addition modules in the systolic array that perform Fourier transform operation on the same data.

7. The AI ​​accelerator according to claim 6, wherein: The complex multiplier comprises: a second selection submodule and a multiplication submodule; wherein: The second selection submodule is adapted to select the second multiplier and the third multiplier of the multiplication submodule; the second multiplier and the third multiplier are preset fixed values, or are output data of the previous multiplication and addition module; The multiplication submodule is adapted to perform a multiplication operation on the second multiplier selected by the second selection submodule.

8. The AI ​​accelerator according to any one of claims 3 to 7, wherein: The operation identified by the operation identification information is a fast Fourier transform operation, and the operation parameter information includes: sampling point number information and cardinality information.

9. The AI ​​accelerator according to claim 8, wherein: The data to be processed is a time domain signal; the data processing unit is suitable for sampling the time domain signal conversion according to the sampling point number information to obtain each sampling value, and converting each sampling value into a corresponding complex matrix according to the cardinality information, and using the complex matrix corresponding to the sampling value as the first operation data.

10. The AI ​​accelerator according to claim 9, wherein: The elements of two adjacent rows in the same column of the complex matrix are real data and imaginary data of the same sampling value, or the elements of two adjacent columns in the same row of the complex matrix are real data and imaginary data of the same sampling value.

11. The AI ​​accelerator according to claim 8, wherein: The other operation data are Fourier transform operation coefficients corresponding to the operation parameter information, or are rotation factors.

12. The AI ​​accelerator according to claim 11, wherein: The control unit is adapted to pre-store the Fourier transform operation coefficients in the Fourier transform operator unit, determine a process of a fast Fourier transform operation based on the sampling point number information and the cardinality information, and adjust the inputs of the first input terminal and the second input terminal of the operation unit according to the determined process of the fast Fourier transform operation; The process of the fast Fourier transform operation includes the number of times the Fourier transform operation is performed, the number of times the rotation factor is updated, and the order of performing the Fourier transform operation and the complex multiplication operation.

13. The AI ​​accelerator according to any one of claims 3 to 7, characterized in that: The operation operation identified by the operation identification information is a convolution operation, and the operation parameter information includes: size information of the convolution kernel.

14. The AI ​​accelerator according to claim 13, wherein: The data to be processed is input feature data; the data processing unit is suitable for converting the input feature data into a three-dimensional matrix corresponding to the operation unit.

15. The AI ​​accelerator according to claim 13, wherein: The other operation data is the one-dimensional convolution kernel obtained after the convolution kernel is split.

16. The AI ​​accelerator according to claim 3, wherein: The operation unit further includes: at least one delay subunit.

17. An electronic device, characterized in that: An AI accelerator comprising any one of claims 1 to 16.

Images