Pointwise convolution computing apparatus and method

Abstract

The application provides a Pointwise convolution calculation device and method, a data cache unit divides a data matrix to obtain a data sub-matrix; a data transmission unit sends the data sub-matrix into a Pointwise convolution calculation core in parallel according to a period, and determines whether to send an intermediate operation result according to a calculation task division condition; a control unit controls the Pointwise convolution calculation core to read a weight value of a current period from a circular shift and perform convolution operation on the sent feature map data; and a data router outputs the convolution operation result to an external memory or the data transmission unit through a bus interface under the control of the control unit. The application optimizes the storage mode and calculation strategy of the feature map data and the weight value data input into the Pointwise convolution calculation core, realizes full reuse and parallel calculation of the data read from the external storage, and greatly reduces the memory access delay and power consumption caused by the access to the external memory in the inference process.

Description

Technical Field

[0001] This invention belongs to the field of chip data computing and processing technology, specifically relating to a pointwise convolution computing device and method. Background Technology

[0002] Hardware chips contain multiple processing chips that handle data through neural networks. The inference process of neural networks generates a large number of matrix operations, requiring significant computational resources and time. To enable the deployment of neural network models, dedicated neural network accelerators significantly improve the computational speed and energy efficiency of neural networks through customized hardware design and algorithm optimization.

[0003] Pointwise convolution is a lightweight convolution method that uses a 1x1 convolution kernel to achieve information exchange between feature map channels. However, PW convolution computation involves a large number of matrix multiplication operations, requiring the reading of necessary weights and feature map data from external memory and the writing of intermediate results back to memory, resulting in a significant waste of on-chip resources and high memory access power consumption and latency. Because pointwise convolution computation involves various channel transformations and multiplication operations of different scales, current computing acceleration platforms lack computational unit optimization and data control strategies specifically for pointwise convolution, limiting the efficiency and speed of pointwise convolution computation. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention provides a Pointwise convolution computation device and method. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] This invention provides a pointwise convolution calculation device comprising:

[0006] The data caching unit is used to retrieve the data matrix and related size information of the feature map from the external memory via the bus, and to divide the data matrix into data sub-matrices according to the convolution calculation requirements and computing resources.

[0007] The data transmission unit is used to send the data submatrix into the Pointwise convolution kernel in parallel according to the period, and to decide whether to send the convolution operation result of the previous period to the Pointwise convolution kernel according to the division of convolution operation tasks.

[0008] The control unit generates control signals to control the Pointwise convolution kernel to cyclically shift and read the weights for the current period from the weight array;

[0009] Pointwise convolution kernels are used to perform convolution operations on the feature map data fed in parallel in the current cycle and the corresponding weights to obtain the convolution operation results.

[0010] The data router is used to receive the convolution operation results of the Pointwise convolution calculation kernel, and under the control of the control unit, output the convolution operation results to the external memory or data transmission unit through the bus interface;

[0011] Buses and bus interfaces are used to enable information exchange between data cache units, control units, data routers, and external storage.

[0012] This invention provides a pointwise convolution calculation method, implemented using a pointwise convolution calculation device. The pointwise convolution calculation method includes:

[0013] S100, the data acquisition unit retrieves weight array, bias data, input feature map data and size information from external memory;

[0014] S200 divides the feature map data matrix into multiple data sub-matrices based on the size of the Pointwise convolution kernel.

[0015] S300: The control unit preloads the weight array into the weight memory and broadcasts the corresponding weights to the Pointwise convolution kernel when the Pointwise convolution kernel needs to perform calculations.

[0016] S400, the data transmission unit inputs the data submatrices into the Pointwise convolution calculation kernel in parallel according to the period;

[0017] S500, the Pointwise convolution kernel performs convolution operations between parallel input data and corresponding weights to obtain the calculation result;

[0018] The S600 Pointwise convolution kernel outputs the convolution operation results through the bus interface.

[0019] This invention proposes a Pointwise convolution computation device and method. By optimizing the architecture and data flow of the Pointwise convolution computation kernel, it achieves the reuse of input feature value data and weight data during the Pointwise convolution operation. Compared with the prior art, this invention achieves feature map data reuse through feature map data flow arrangement and shift circuits within the PE array. Data is loaded only once during the computation process, reducing data loading time and improving the utilization of computation units. Furthermore, it achieves weight broadcasting during the computation process through weight storage strategy optimization and input feature map data flow arrangement, improving computational parallelism. The addition chain composed of internal PE units enables caching and summing of intermediate value data in the Pointwise computation, avoiding additional storage overhead. This invention designs a computation method based on this Pointwise convolution computation device, utilizing control units and data routers to configure multiple computation arrays in the computation kernel, achieving computational reconfiguration and ensuring the versatility of the computation device.

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] Figure 1 A schematic diagram of a Pointwise convolution calculation device provided in an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a PE unit structure provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a calculation method based on a Pointwise convolution calculation device provided in an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of input feature map data partitioning provided in an embodiment of the present invention;

[0025] Figure 5 This is a schematic diagram of Pointwise convolution weight partitioning provided in an embodiment of the present invention;

[0026] Figure 6 This is a schematic diagram of the Pointwise weight storage space arrangement provided in an embodiment of the present invention;

[0027] Figure 7 This is a schematic diagram of the timing arrangement of input feature map data provided in an embodiment of the present invention;

[0028] Figure 8-12 This is a schematic diagram of the Pointwise convolution calculation process of the PE array provided in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0030] Please see Figure 1 , Figure 1 A schematic diagram of a Pointwise convolution calculation device provided in an embodiment of the present invention includes: a data cache unit, a data transmission unit, a control unit, a data router, a Pointwise convolution calculation kernel, and a bus and bus interface for information interaction with external memory.

[0031] The data caching unit is used to retrieve the data matrix and related size information of the feature map from the external memory via the bus, and to divide the data matrix into data sub-matrices according to the convolution calculation requirements and computing resources.

[0032] The data transmission unit is used to periodically send the data submatrix into the Pointwise convolution kernel in parallel, and to decide whether to send the intermediate calculation results to the Pointwise convolution kernel according to the division of convolution operation tasks.

[0033] It is worth noting that the data transmission unit reads and sends intermediate data of the Pointwise convolution kernel input from the data router. If the feature map data of the last cycle of the data matrix completes the convolution operation, the entire feature map convolution process is complete. If it is not the last cycle, the Pointwise convolution kernel requires the convolution result of the previous cycle to participate in the convolution operation. In this case, the data transmission unit needs to read the convolution result of the previous cycle and use it as an intermediate result to participate in the next convolution operation until the entire data matrix operation is completed.

[0034] The control unit generates control signals to control the Pointwise convolution kernel to cyclically shift and read the weights for the current period from the weight array;

[0035] It is worth noting that: before the calculation begins, the weights are stored in the on-chip weight memory after the storage space is arranged by the input control unit, and are read cyclically during the calculation process, and are used to perform parallel calculations of pointwise convolution with the feature map data input to the convolution calculation kernel through the data transmission unit.

[0036] The control unit is used to receive the weight array in advance via the bus result, read the weight of the current period from the weight array by cyclic shifting, and concatenate and rearrange the weights of the current period to form a weight matrix;

[0037] Pointwise convolution kernels are used to perform convolution operations on the feature map data fed in parallel in the current cycle and the corresponding weights to obtain the convolution operation results.

[0038] The data submatrix is ​​fed in periodically, and the data fed in each period is called feature map data.

[0039] The pointwise convolution kernel is used to repeat the process of treating each cycle as the current cycle. It performs convolution operations based on the convolution operation results of the previous cycle, the feature map data in the current cycle, and the corresponding weights, and transmits the convolution operation results of the current cycle to the data router until the convolution operation of the data matrix is ​​completed.

[0040] The data router is used to receive the convolution operation results of the Pointwise convolution calculation kernel, and under the control of the control unit, output the convolution operation results to the external memory or data transmission unit through the bus interface;

[0041] Buses and bus interfaces are used to enable information exchange between data cache units, control units, data routers, and external storage.

[0042] Continue to refer to Figure 1 As an optional embodiment of the present invention, the Pointwise convolutional kernel includes eight parallel PE computation arrays. Each PE computation array, in cooperation with a data caching unit, a data transmission unit, a control unit, and a data router, performs convolution operations on the feature map data of each cycle with the corresponding weights.

[0043] It is worth noting that for Pointwise convolution computation, the data matrix can sometimes be computed using only the proposed PE array once to obtain the final target output result; however, it is not possible to obtain the final result using only one PE array, and multiple PE arrays are required to participate in the computation. This is related to the number of convolution input channels in the current Pointwise convolution computation task.

[0044] After the input feature maps are time-sequentially arranged (the next row is entered one cycle later than the current row), they are horizontally input, shifted in the corresponding PE rows, and then multiplied and accumulated with the weight data read in the cyclic process within the PE. The current PE array calculation ends when all input feature map data from the right side of the array is completely shifted out from the left side of the array. The final target result or intermediate operation result generated by the convolution calculation is propagated vertically. Whether the data output to the router from the last row of PE units is the final target result or an intermediate operation result depends on the pre-defined task division (i.e., the number of input channels of the current Pointwise convolution; if the current number of Pointwise convolution channels is n, then the target output data needs to be processed by n PEs, thus requiring n vertical propagations). If the calculation can be completed in one PE array (when n is less than or equal to 8), then a single PE array can be used. If the calculation cannot be completed in one PE array, then multiple PE arrays are needed for the calculation. For example, if the current Pointwise convolution input channels are 16, then obtaining the target calculation result requires the cooperation of two PE arrays. The data output from the first PE array to the data router is the intermediate calculation result, which is then passed through the data router to the second PE array to continue participating in the calculation.

[0045] For example, in a multiply-add summation operation, sometimes only the multiply-add summation of 8 input feature map data and 8 corresponding weights is required. In this case, a single PE array can meet the computational needs. The result output from the 8th row of PE cells to the data router is the target result, which is then output through bus interface 3. However, if a pointwise convolution operation with 16 input channels is to be implemented, the data needs to be processed through two PE arrays. The data output from the 8th row of PE cells in the first PE array to the data router is the intermediate operation result. Because the current output result is only the multiply-add summation of 8 input feature map data and 8 corresponding weights, this intermediate operation result needs to be transmitted through the data router to another PE array as the initial bias input to the first row of PE cells in the array to continue the operation. The output of the last row of PE cells in this second PE array completes the multiply-add summation of 16 input feature map data and 16 corresponding weights.

[0046] Please see Figure 2 As an optional embodiment of the present invention, each PE computing array is composed of 8*8 isomorphic PE units;

[0047] Each PE unit includes a weight input interface, a feature map data input interface, a feature map data output interface, an upper PE operation result input interface, a PE operation result output interface, a feature map data register, an operation result register, a multiplier, and an adder;

[0048] Each PE unit interacts with other units during convolution operations via a data interface, performs data operations through multipliers and adders, and uses registers for data caching. The weight input interface is used to input the weights participating in the Pointwise convolution operation. The feature map input interface is used to cache feature map data in a feature map data register. The feature map output interface is used to output the feature map data stored in the feature map data register. The feature map data register is used to cache input feature map data. The multiplier multiplies the feature map data with the corresponding weights and inputs the multiplication result to the adder. Following the same column from top to bottom, the operation result input interface of the upper PE unit is used to input the intermediate value data from the upper PE unit as the adder input. The adder sums the multiplier operation result with the intermediate value data from the upper PE unit. The operation result register caches the adder's summation result. The PE operation result output interface outputs the operation result of the current PE unit cached in the result register to the next PE unit in the same column.

[0049] For example, if the desired final result is the sum of 16 numbers multiplied and added together, the PE unit performs the multiplication and addition operation. That is, in each cycle, the input weights are multiplied by the feature map data, and then added to the intermediate calculation results obtained earlier. After 16 cycles, the multiplication and accumulation operation of 16 numbers can be completed. The result obtained at this time is the target result data, and all the intermediate calculation results are the intermediate value data.

[0050] Please see Figure 2 As an optional embodiment of the present invention, each PE unit in the PE computing array, under the control of the control unit, transmits the feature map data cached in the PE unit to the PE unit to the left of the row where the PE unit is located.

[0051] The PE operation result input interface above the first row of PE cells in the PE computation array is used to input the preset initial bias; the result output interface of the last row of PE cells is used to output the convolution operation result of the PE array to the data router or to continue to participate in the operation as intermediate value data; the feature map input interface of the rightmost column of PE cells is used to input feature map data.

[0052] Please see Figure 2 As an optional embodiment of the present invention, the control unit is used to receive the weight array through the bus interface, read the weights of the weight array by cyclic shifting, and rearrange the read weights according to the PE cell positions to obtain the weight matrix; and load the weight matrix into the weight memory;

[0053] Under the control of the clock signal, the pointwise convolution kernel reads the corresponding weight data from the weight memory and broadcasts it to the corresponding row PE unit. The weight input interface of the PE unit in the same row of each PE calculation array is connected to the weight output interface of the same weight memory, and they share the same weight.

[0054] It is worth noting that the PE array obtains weight data from the on-chip weight memory to perform pointwise convolution operations. PE cells in the same row share the same weight data. The weight data in the weight register is concatenated and loaded before the pointwise calculation begins. During the calculation, the data is read and split within one clock cycle according to the storage address and broadcast to the corresponding PE calculation cell column.

[0055] Please see Figure 2 As an optional embodiment of the present invention, the PE units in the same column of the PE array are connected to form an adder chain through the PE operation result input interface and the PE operation result output interface, so as to realize the data accumulation of the PE computing array for convolution operation, and output the operation result of the PE array to the data router through the operation result output interface of the last-stage PE unit.

[0056] As an optional embodiment of the present invention, the Pointwise convolution calculation device is implemented by an FPGA or ASIC as the computation part in a neural network accelerator.

[0057] Please see Figure 3 , Figure 3 A schematic flowchart of a calculation method based on a Pointwise convolution calculation device provided in this embodiment of the invention includes the following steps:

[0058] S100, the data acquisition unit retrieves weight array, bias data, input feature map data and size information from external memory;

[0059] S200 divides the feature map data matrix into multiple data sub-matrices based on the size of the Pointwise convolution kernel.

[0060] S300: The control unit preloads the weight array into the weight memory and broadcasts the corresponding weights to the Pointwise convolution kernel when the Pointwise convolution kernel needs to perform calculations.

[0061] S400, the data transmission unit inputs the data submatrices into the Pointwise convolution calculation kernel in parallel according to the period;

[0062] S500, the Pointwise convolution kernel performs convolution operations between parallel input data and corresponding weights to obtain the calculation result;

[0063] The S600 Pointwise convolution kernel outputs the convolution operation results through the bus interface.

[0064] As an optional embodiment of the present invention, the Pointwise convolution kernel is used for:

[0065] Parallel reception of feature map data sent by the data transmission unit in the current cycle;

[0066] Among them, the feature map data of all periods constitute the feature map data matrix;

[0067] Under the control of the clock signal, the weights corresponding to the feature map data of the current period are read from the weight matrix; wherein, the weight matrix is ​​formed by cyclically shifting the weights of the current period from the weight array and concatenating and rearranging the weights of the current period.

[0068] The process of performing convolution operations on the data matrix is ​​repeated for each period, taking the convolution operation result from the previous period, the feature map data in the current period, and the corresponding weights, and then transmitting the convolution operation result of the current period out, until the convolution operation of the data matrix is ​​completed.

[0069] The following describes a convolution calculation method based on a Pointwise convolution calculation device disclosed in this application, with specific examples.

[0070] First, the data required for calculation is obtained. Suppose that the number of channels in the input feature map matrix is ​​16 and the dimension of the feature map matrix is ​​8*8*16. After pointwise convolution, the number of channels in the feature map is 16. Then the dimension of the weight matrix is ​​16*16 and the channel bias is 16*1. All relevant data are obtained from external memory through the bus.

[0071] Secondly, to ensure efficient pointwise convolution processing, the input data needs to be partitioned and computational tasks allocated. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is a schematic diagram of input feature map data partitioning provided in an embodiment of the present invention. The preset number of PE arrays in the Pointwise convolution kernel is 8. Each PE array is composed of 8*8 isomorphic PE units. Therefore, the maximum number of input feature map channels that the PE array can process is 8. The input feature map data is divided into two sub-matrices according to the number of channels in units of 8 channels. At the same time, the non-channel dimension is reduced to 1. At this time, the size of the sub-matrix is ​​8*64. Pointwise convolution operation is performed by the two PE arrays in the kernel.

[0072] Next, before computation begins, the weights need to be input into the network to configure the computation array. Please refer to [link / reference]. Figure 5 , Figure 5This is a schematic diagram of Pointwise convolution weight partitioning provided in an embodiment of the present invention. The input weight matrix is ​​divided into four 8*8 weight sub-matrices according to the size of the PE array. Before the calculation starts, the weight memory of the PE array is configured according to the calculation requirements. During the configuration process, the storage space in the weight memory is optimized by weight splicing.

[0073] Please see Figure 6 , Figure 6 This is a schematic diagram of the Pointwise weight storage space arrangement provided in an embodiment of the present invention. Figure 5 The weight matrix (0, 0) obtained by dividing the matrix is ​​used as an example. The 8*8 weight matrix is ​​regrouped and spliced ​​along the secondary diagonal direction and stored in 8 storage spaces in the weight memory. During the operation, 8 weight data can be read from the weight memory in one cycle through a single storage address and transmitted to each row of PE cells in the PE array through 8 weight data lines.

[0074] Please see Figure 7 , Figure 7 This is a schematic diagram of the timing arrangement of input feature map data provided in an embodiment of the present invention. During the calculation process, 8 channels of input feature map data are input into the PE array in parallel through the feature map data input interface of the rightmost PE unit of each row. The input timing of each row of input data is one cycle apart under the control of the data transmission module. Taking an input feature map data matrix with a size of 8*64 as an example, the second row of input data is one cycle behind the first row of data. The corresponding delay can be equivalently achieved by inserting data 0. The transmission of the input feature map data matrix requires a total of 72 clock cycles.

[0075] Please see Figure 8-12 , Figures 8 to 11 This is a schematic diagram of the PE array pointwise convolution calculation process provided in an embodiment of the present invention, illustrating the calculation flow of the pointwise convolution calculation device and method. During the calculation process, feature map data is presented in... Figure 7 The timing input PE array shown is shifted through a register chain formed within the array; the weight data is... Figure 6As shown, the weights are pre-stored in the weight register before the calculation begins, and are read by cyclic shifting according to their addresses during the calculation. The rule for cyclic shifting is to shift cyclically along the lower diagonal, which is the diagonal formed from the upper right corner to the lower left corner. The PW calculation solves the partial sums through the multipliers inside the PE units in the same column, adds the partial sums through the chain of adders, and outputs the calculation results of the last row of PE units to the data router. The calculation results are output in the 8th clock cycle, and all PE units work in the 15th clock cycle. During the calculation, all input feature map data is read only once, and the weight data in the weight memory is read and used cyclically. During the calculation process in the PE array, there is no delay caused by the preparation of calculation data and the storage of intermediate calculation results, and the parallel calculation of PW convolution by 64 PE units is realized, which greatly saves the calculation time.

[0076] This invention proposes a Pointwise convolution computation device and method. By optimizing the architecture and data flow of the Pointwise convolution computation kernel, it achieves the reuse of input feature value data and weight data during the Pointwise convolution operation. Compared with the prior art, this invention achieves feature map data reuse through feature map data flow arrangement and shift circuits within the PE array. Data is loaded only once during the computation process, reducing data loading time and improving the utilization of computation units. Furthermore, it achieves weight broadcasting during the computation process through weight storage strategy optimization and input feature map data flow arrangement, improving computational parallelism. The addition chain composed of internal PE units enables caching and summing of intermediate value data in the Pointwise computation, avoiding additional storage overhead. This invention designs a computation method based on this Pointwise convolution computation device, utilizing control units and data routers to configure multiple computation arrays in the computation kernel, achieving computational reconfiguration and ensuring the versatility of the computation device.

[0077] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0078] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0079] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A Pointwise convolution computing apparatus, characterized by, Comprise: The data cache unit is used to obtain the data matrix of the feature map and the related size information from the external memory through the bus, and divide the data matrix to obtain the data sub-matrix according to the convolution calculation requirement and the calculation resource; The data transmission unit is used to send the data sub-matrix into the Pointwise convolution calculation core in parallel according to the period, and decide whether to send the convolution operation result of the last period into the Pointwise convolution calculation core according to the convolution operation task division; The control unit is used to generate a control signal to control the Pointwise convolution calculation core to cyclically shift and read the weight value of the current period from the weight array; The Pointwise convolution calculation core is used to perform convolution operation on the feature map data and the corresponding weight value sent in parallel in the current period to obtain the convolution operation result; the Pointwise convolution calculation core comprises 8 parallel PE calculation arrays, each PE calculation array is arranged by 8*8 isomorphic PE units; each PE unit in the PE calculation array transmits the feature map data cached in the PE unit to the left PE unit of the row where the PE unit is located under the control of the control unit; the PE units in the same column in the PE array are connected by the PE operation result input interface and the PE operation result output interface to form an adder chain, so as to realize data accumulation of the convolution operation of the PE calculation array, and output the operation result of the PE array to the data router through the operation result output interface of the last PE unit; The data router is used to receive the convolution operation result of the Pointwise convolution calculation core, and output the convolution operation result to the external memory or the data transmission unit through the bus interface under the control of the control unit; The bus and the bus interface are used to realize information interaction between the data cache unit, the control unit, the data router and the external memory.

2. The Pointwise convolution calculation device according to claim 1, wherein The control unit is used to receive the weight array through the bus in advance, cyclically shift and read the weight value of the current period from the weight array, and splice and rearrange the weight value of the current period to form a weight matrix; The Pointwise convolution calculation core is used to repeat each period as the current period, perform convolution operation according to the convolution operation result of the last period, the feature map data in the current period and the corresponding convolution weight value, and transmit the convolution operation result of the current period to the data router until the convolution operation of the data matrix is completed.

3. The Pointwise convolution computing apparatus of claim 2, wherein, Each PE calculation array performs convolution operation on the feature map data and the corresponding weight value in each period under the cooperation of the data cache unit, the data transmission unit, the control unit and the data router.

4. The Pointwise convolution computing apparatus of claim 3, wherein, Each PE unit comprises a weight input interface, a feature map data input interface, a feature map data output interface, an upper PE operation result input interface, a PE operation result output interface, a feature map data register, an operation result register, a multiplier and an adder; Each PE unit exchanges information through a data interface during convolution operation, performs data operation through a multiplier and an adder, and realizes data caching through a register; the weight input interface is used for inputting weights participating in Pointwise convolution operation; the feature map input interface is used for caching feature map data into a feature map data register; the feature map output interface is used for outputting feature map data stored in the feature map data register; the feature map data register is used for caching input feature map data; the multiplier is used for multiplying feature map data and corresponding weights, and inputting the multiplication result into the adder; the operation result input interface of the PE unit above is used for inputting intermediate value data of the PE unit above as an adder input in the same column from top to bottom; The adder is used for adding and summing the multiplication result of the multiplier and the intermediate value data of the PE unit above; the operation result register is used for caching the summing result of the adder; and the PE operation result output interface is used for outputting the operation result of the current PE unit cached in the result register to the PE unit in the same column in the next row.

5. The Pointwise convolution computing apparatus of claim 4, wherein, The operation result input interface of the PE unit above in the first row of the PE computing array is used for inputting a preset initial bias; the result output interface of the PE unit in the last row is used for outputting the convolution operation result of the PE array to a data router or as intermediate data to continue participating in operation; and the feature map input interface of the PE unit in the rightmost column is used for inputting feature map data.

6. The Pointwise convolution computing apparatus of claim 5, wherein, The control unit is used for receiving a weight array through a bus interface, reading weights of the weight array in a cyclic shift manner, splicing and rearranging the read weights according to PE unit positions to obtain a weight matrix, and loading the weight matrix into a weight storage. The Pointwise convolution computing core reads corresponding weight data from the weight storage under the control of a clock signal and broadcasts the weight data to corresponding row PE units; the weight input interface of the PE unit in each PE computing array is connected with the weight output interface of the same weight storage, and shares the same weight.

7. The Pointwise convolution computing apparatus of claim 1, wherein, The Pointwise convolution computing device is realized by FPGA or ASIC, and serves as a computing part in a neural network accelerator.

8. A method of pointwise convolution computation, characterized in that, The Pointwise convolution computing method is realized by using the Pointwise convolution computing device in any one of claims 1 to 7, and includes the following steps: S100, a data acquisition unit acquires a weight array, bias data, input feature map data and size information from an external storage; S200, a feature map data matrix is divided into a plurality of data sub-matrices according to the size of the Pointwise convolution computing core; S300, a control unit preloads a weight array into a weight storage, and broadcasts corresponding weights to the Pointwise convolution computing core when the Pointwise convolution computing core needs to operate; S400, a data transmission unit inputs data sub-matrices into the Pointwise convolution computing core in a cycle; S500, the Pointwise convolution computing core performs convolution operation on the input data and corresponding weights to obtain a calculation result; and S600, a data output unit outputs the calculation result to an external storage. S600, the Pointwise convolution calculation core outputs the convolution operation result through the bus interface.

9. The Pointwise convolution calculation method of claim 8, wherein, The Pointwise convolution calculation core is used for: The parallel receiving data transmission unit receives the feature map data sent by the current cycle; Wherein, the feature map data of all cycles constitute a data matrix of the feature map; Under the control of the clock signal, the weight value corresponding to the feature map data of the current cycle is read from the weight value matrix; wherein, the weight value matrix is obtained by cyclically shifting the weight value array to read the weight value of the current cycle, and then splicing and rearranging the weight value of the current cycle to form the weight value matrix; Each cycle is taken as the current cycle to repeat the process of performing convolution operation according to the convolution operation result of the previous cycle, the feature map data in the current cycle and the corresponding weight value, and transmitting the convolution operation result of the current cycle, until the convolution operation of the data matrix is completed.

Images