A method and device for implementing depth-wise convolution calculation by fully multiplexing GEMM and ALU architecture

Abstract

The application discloses a kind of completely multiplex GEMM and ALU architecture implementation Depth-wise convolution's computing method and device, comprising: determining GEMM and ALU architecture, obtains the input feature map data and weight data of convolution layer, input feature map data and weight data are divided into several data blocks and sub data block, and the calculation order of each data block is obtained;According to the calculation order, the data of different channels of each input feature map data block and each weight sub data block is input to the corresponding single-channel computing unit;Each single-channel computing unit carries out convolution calculation to each input feature map data matrix and each weight sub data matrix, obtains output feature map data block;Output feature map data block is written back in the corresponding address of memory according to sequence.This application improves the utilization of GEMM computing resources in Depth-wise convolution operation process, improves the throughput and processing performance of data.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, and in particular relates to a computational method and apparatus for implementing depth-wise convolution by fully reusing the GEMM and ALU architecture. Background Technology

[0002] Convolutional Neural Networks (CNNs) extract features from input data through convolution operations, thereby achieving data recognition and classification, and have achieved excellent results in applications such as image classification and object detection. In CNNs, the standard convolutional layer is primarily responsible for the crucial task of feature extraction. The input data of a standard convolutional layer consists of two parts: feature map data with multiple channels and multiple convolutional kernels with multiple channels. As the application of deep learning expands across industries, the use of CNNs in mobile and edge computing scenarios is becoming increasingly common. However, due to the limitations of storage bandwidth and computing resources in mobile and edge computing scenarios, the effectiveness of CNNs is not significant. Therefore, some neural network models have proposed Depth-wise convolutional layers, which have fewer parameters and lower computational complexity. The input data of a Depth-wise convolutional layer consists of multi-channel feature map data and a multi-channel convolutional kernel. The channels of the feature map data and the convolutional kernel correspond to each other, with each channel of the convolutional kernel responsible for one channel of the feature map data.

[0003] Standard convolution requires accumulating the convolution results of all channels. A common implementation method is to use GEMM (general matrix-vector multiplication) for acceleration, performing convolution calculations in parallel on the 3D input, with both channel-parallel and kernel-parallel approaches. Accelerators that use multiply-accumulate units as the smallest unit for convolution calculations achieve both channel-parallel and kernel-parallel approaches through the regular combination and placement of these units. Depth-wise convolutional layers have only one kernel and perform calculations channel-by-channel. Therefore, there is no possibility of kernel-parallel approaches in depth-wise convolution. Depth-wise convolutional layers in GEMM have only an n / (n*m) utilization rate, where n is the number of parallel channels, m is the number of parallel kernels, and n*m is the number of multiply-accumulate units in GEMM.

[0004] In existing products, Arm Ethos supports both standard convolution and depth-wise convolution. When computing standard convolution, Arm Ethos iterates and outputs feature maps in the order of dimensional direction, horizontal direction, and vertical direction, reusing the input feature map in the dimensional direction. When computing depth-wise convolutional layers, it iterates the input and output feature maps in the order of dimensional direction, horizontal direction, and vertical direction, but does not reuse the input feature map. Therefore, Arm Ethos only performs channel parallelism when implementing depth-wise convolution. Before each convolution calculation, it needs to read the input feature map and convolution kernel data for the number of parallel channels from memory. Compared to the acceleration effect of ordinary convolution, the execution efficiency of depth-wise convolution is relatively low.

[0005] Some accelerators for depth-wise convolutional layers are designed solely based on the characteristics of depth-wise convolution, sacrificing support for standard convolutional layers. This requires adding two extra accelerators—one for standard convolution and another for depth-wise convolution—increasing the area required and hindering implementation on mobile and edge devices. Some depth-wise convolutional layer accelerators use the `img2col` operation to convert the input feature map and weight data into matrices suitable for GEMM operations before performing convolution calculations. This approach increases the matrix transformation overhead of `img2col` and fails to adequately meet the low-power requirements of mobile and edge devices. Summary of the Invention

[0006] The purpose of this invention is to provide a computational method and apparatus for implementing depth-wise convolution by fully reusing the GEMM and ALU architectures, so as to solve the above-mentioned technical problems.

[0007] To address the aforementioned technical problems, the present invention provides a specific technical solution for a method and apparatus for implementing depth-wise convolution that fully reuses the GEMM and ALU architectures, as follows: A method for calculating depth-wise convolutions that fully reuses the GEMM and ALU architectures, the method comprising: Step 1: Determine the GEMM and ALU architectures, determine the number and arrangement of multiply-accumulate units in the GEMM architecture, and determine the number of multiply-accumulate units in the ALU architecture; Step 2: Obtain the input feature map data and weight data of the convolutional layer, divide the input feature map data and weight data into several data blocks and sub-data blocks, and obtain the calculation order of each data block; Step 3: According to the calculation order, input the data from different channels of each input feature map data block and each weight sub-data block into the corresponding single-channel calculation unit; each single-channel calculation unit performs convolution calculation on each input feature map data matrix and each weight sub-data matrix to obtain the output feature map data block; Step 4: Write the output feature map data blocks back to the corresponding addresses in memory according to the calculation order.

[0008] Furthermore, step 1 includes the following specific steps: Analyzing the number of channels in the input feature map of the standard convolutional layer, the number of convolutional kernels, and the requirements of the application scenario, the preset number of channels parallel and the preset number of convolutional kernels parallel are determined respectively. Based on the preset number of channels parallel, the number of single-channel computation units in the GEMM architecture is determined. Based on the preset number of convolutional kernels parallel, the number of multiply-accumulate units in each single-channel computation unit in the GEMM architecture and the number of multiply-accumulate units in the ALU architecture are determined. The single-channel computation units of GEMM are reused as input channels parallel for Depth-wise convolution. At the same time, the multiply-accumulate units in each single-channel computation unit are reused as convolution computation units for the input feature map data and weight data in each channel of Depth-wise convolution.

[0009] Furthermore, in step 1, for the standard convolutional layer using GEMM, the acceleration method is to process 8 convolutional kernels in parallel. Each convolutional kernel performs convolution calculations in parallel according to 8 channels. GEMM outputs the calculation results according to the output channel in each calculation cycle, resulting in a total of 8 output feature map data. For the depth-wise convolutional layer that reuses the GEMM architecture, the acceleration method is to process 8 channels in parallel. The 8 multiply-accumulators of each single-channel calculation unit perform convolution calculations on the 8 input feature map data and 8 weight data of each channel. GEMM outputs the calculation results according to the output channel in each calculation cycle, resulting in a total of 8 output feature map data.

[0010] Furthermore, step 2 includes the following specific steps: The input feature map data of the standard convolutional layer and the depth-wise convolutional layer are divided into channels by a predetermined number of channels for parallel processing, resulting in input feature map data blocks. Each input feature map data block has the same channel dimension, which is equal to the predetermined number of channels for parallel processing. The weight data of the standard convolutional layer and the depth-wise convolutional layer are also divided into channels by a predetermined number of channels for parallel processing, resulting in weight data blocks. Each weight data block has the same channel dimension, which is equal to the predetermined number of channels for parallel processing. If the channel dimension of the input feature map data and the weight data is less than an integer multiple of the predetermined number, the number of channels in the input feature map and the weight data are adjusted to an integer multiple of the predetermined number by padding the channels with zeros. Then, the adjusted input feature map data and the weight data are divided into data blocks according to the same steps. Based on the division method, the position index of the input feature map data block in the input feature map and the position index of the weight data block in the convolutional kernel are determined respectively.

[0011] Furthermore, in step 2, the weight data block of the Depth-wise convolutional layer is further divided into sub-weight data blocks of a preset size in the row and column dimensions. The preset size is determined according to the number of multiply-accumulators in each single-channel computing unit of GEMM. When the weight data block is less than an integer multiple of the preset size, the row and column dimensions of the weight data block are adjusted to an integer multiple of the preset size by padding with zeros in the rows and columns. Then, the adjusted weight data block is divided into sub-weight data blocks in the same way. Based on the convolution calculation loop of standard convolutional layers and depth-wise convolutional layers, as well as the position index of each input feature map data block in the input feature map and the position index of each sub-weight data block in the convolution kernel, the convolution calculation order of the input feature map data blocks and each sub-weight data block of each convolutional layer is determined.

[0012] Furthermore, step 3 includes the following specific steps: GEMM sequentially performs convolution calculations on the input feature map data blocks and sub-weight data blocks of each computation cycle to obtain the corresponding output feature map data blocks. Standard convolution in GEMM adopts a parallel approach with both channel parallelism and kernel parallelism. Specifically, the single-channel computation unit of GEMM implements kernel parallelism for standard convolution, and the multiply-accumulate units in each single-channel computation unit implement channel parallelism. Depth-wise convolution in GEMM adopts a channel parallel approach, with each channel performing Depth-wise convolution calculations independently. The single-channel computation unit of GEMM implements channel parallelism for Depth-wise convolution, and the multiply-accumulate units in each single-channel computation unit obtain independent convolution sums for each channel. If the size of the sub-weight data block of the Depth-wise convolution layer is larger than the number of multiply-accumulate units in a single-channel computation unit, the sub-weight data block needs to be split when reusing GEMM to implement Depth-wise convolution, and the convolution calculation of the input feature map data and weight data needs to be completed in two computation cycles.

[0013] Furthermore, in step 3, when processing the standard convolutional layer, the input of the GEMM architecture of this application is an input feature map data block and a weight data block. The data in each weight data block comes from the convolutional kernels processed in parallel. The number of convolutional kernels processed in parallel is equal to the preset number of single-channel computing units. The number of channels of each input feature map data block and sub-weight data block processed by GEMM in each cycle is equal to the number of multiply-accumulate units in a single-channel computing unit. When processing the depth-wise convolutional layer, the input of the GEMM and ALU architecture is an input feature map data block and a sub-weight data block. The number of channels of each input feature map data block and each sub-weight data block is equal to the number of single-channel computing units. The number of input feature map data and weight data processed by each single-channel computing unit in each computing cycle of GEMM is less than or equal to the number of multiply-accumulate units in a single-channel computing unit.

[0014] Furthermore, in step 3, the GEMM and ALU architecture computes depth-wise convolution using a four-layer loop. During this process, the first layer divides the input feature map into a predetermined number of groups (i.e., the number of input feature map data blocks) in parallel channel-wise manner, and the output feature map data is also output in parallel channel-wise manner. The second loop divides the weight data blocks into groups of a predetermined size (i.e., the number of sub-weight data blocks in each weight data block). Therefore, for the next three loops, depth-wise convolution is implemented using weight reuse, with each sub-weight data block corresponding to a partial sum of an output feature map data block. The third and fourth layers define the length and width of the output feature map data blocks, respectively. GEMM obtains the intermediate results of the output feature map row by row and writes them to a dedicated storage layer. In the on-chip memory storing intermediate results of the output feature map, while calculating the output feature map result, the intermediate result at the same position as the current output feature map data is read from the on-chip buffer, and added to the convolution sum obtained in the current calculation cycle. After obtaining the new intermediate result, it is written back to the on-chip memory at the same address as when it was read. If the size of the sub-weight data block of the depth-wise convolutional layer is greater than the number of multiply-accumulate units in a single-channel computing unit, the sub-weight data block is split into two parts. The number of weight data in the first part is equal to the number of multiply-accumulate units, and the second part is the remaining weight data. The convolution calculation of the input feature map data and weight data is completed on the GEMM architecture and ALU architecture, respectively. When the second layer loop executes to the last one, the sum is the final data of the output feature map data.

[0015] Further, step 4 includes the following specific steps: According to the description of the four-layer loop in step 3, after the last three loops, an output feature data block is obtained. The size of this output feature map data block is equal to the parameters in the third and fourth loops. There are two cases for the calculation of the output feature map data: the first case is that the size of the weight data block is less than or equal to the preset size, and the second case is that the size of the weight data block is greater than the preset size. If the size of the weight data block is less than or equal to the preset size, there is only one weight sub-data block, and the second loop will only be executed once. After the third and fourth loops are completed, the result is the final output feature map data block. According to the position index of the current output data block in the entire output feature map data and the base address of the output data in the on-chip cache, the starting address of the output feature map data block in the on-chip cache is determined, and then the output feature map data block is written into the on-chip cache. If the size of the weight data block is greater than the preset size, the output feature map data block is calculated. Given a specific size, there will be multiple weighted sub-data blocks. The second loop will execute multiple times. The inner two loops, namely the third and fourth loops, obtain intermediate results of the output feature map. This data is written into a register and participates in the accumulation in subsequent loops. After the second loop ends, all intermediate results with the same index are accumulated to obtain the final output feature map data. The number of channels in the output feature map of the depth-wise convolutional layer is the same as the number of channels in the input feature map. Therefore, the index of the input channel number in the first loop is also the index of the output channel number. According to the index of the current output feature map data block and the base address of the output data in the on-chip buffer, the starting address of the output feature map data block in the on-chip buffer is determined, and the output feature map data block is written back to the on-chip buffer. After the first loop ends, all output feature map data is obtained and written sequentially to the off-chip memory according to the storage format of the output feature map data in the off-chip memory.

[0016] The present invention also discloses a computing device for implementing Depth-wise convolution by fully reusing the GEMM and ALU architectures, comprising: off-chip memory and on-chip cache related to data storage, a data transport module and a Depth-wise computing module related to data computation, wherein the Depth-wise computing module includes the GEMM architecture and the ALU architecture. The GEMM architecture and ALU architecture determine the four-layer loop order of the Depth-wise convolution. The first layer is the number of groups into which the input feature map is divided according to a preset number of channels in parallel, i.e., the number of input feature map data blocks. The output feature map data is also output in a channel-parallel manner. The second layer is the number of groups into which the weight data blocks are divided according to a preset size, i.e., the number of sub-weight data blocks in each weight data block. Therefore, for the next three layers, the weight reuse method is used to implement the Depth-wise convolution. Each sub-weight data block corresponds to a partial sum of an output feature map data block. The third and fourth layers are the length and width of the output feature map data block, respectively. GEMM obtains the intermediate results of the output feature map row by row and writes them into the on-chip memory that is dedicated to storing the intermediate results of the output feature map. Based on these four layers, the transport order and calculation order of the input image data and weight data are determined. According to this transport order, the data transport module transports the input image data and weight data from the off-chip memory to the on-chip buffer. The computation module includes a selector controlled by a computation mode field in the instructions. This field distinguishes between standard convolution and depth-wise convolution. For standard convolution, GEMM and ALU computations are performed independently in channel-parallel mode, determined by the computation type field in the instructions, which also determines the data to be read from the on-chip buffer during computation. For depth-wise convolution, GEMM and ALU computations are performed simultaneously, both using channel-parallel mode. Intermediate results for corresponding channels in the GEMM and ALU architectures are accumulated in the ALU to obtain the final output of the number of parallel channels. Based on the computation order of depth-wise convolution, the computation module determines the data address in the on-chip buffer and reads it into the GEMM and ALU architectures via a data interface. Intermediate results are calculated and accumulated in the GEMM architecture, serving as input data for the ALU architecture. This intermediate result is then added to the convolution results in the ALU to obtain the convolution sum of the number of parallel channels, which is written to the on-chip buffer. The device includes a selector controlled by a mode field in the instructions. Under the control of the first loop, the output results in the on-chip buffer are moved to the off-chip memory through the data transfer module, and the next round of Depth-wise convolution calculation of input image data and weight data begins.

[0017] The computational method and apparatus for implementing depth-wise convolution by fully reusing the GEMM and ALU architectures of the present invention have the following advantages: This invention uses the GEMM and ALU architecture, originally designed to accelerate standard convolutions, to accelerate depth-wise convolutions. It eliminates the need for additional accelerators specifically optimized for depth-wise convolution layers, saving significant area and cost.

[0018] The GEMM and ALU architecture of this invention is based entirely on the optimization and acceleration of standard convolution, and improves the utilization of all computing units when accelerating depth-wise convolution. It ensures that the computing units of GEMM can be efficiently utilized in each computing cycle when performing standard convolution and depth-wise convolution. At the same time, this invention adopts the method of simultaneously accelerating depth-wise convolution using GEMM and ALU architecture, which saves a lot of computing cycles and improves computing efficiency.

[0019] This invention proposes a device that fully reuses the GEMM and ALU architectures, supports both standard convolutional computation mode and depth-wise convolutional computation mode, and distinguishes between the two computation modes, thereby improving the parallelism and utilization of the GEMM and ALU architectures in the depth-wise convolutional computation mode. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the structure of the Depth-wise convolution method implemented by fully reusing the GEMM and ALU architectures of the present invention. Figure 2 This is a schematic diagram of the weighted data segmentation principle of the Depth-wise convolution of the present invention; Figure 3 This is a schematic diagram illustrating the calculation process of Depth-wise convolution in this invention. Figure 4 A module diagram of a device that fully reuses the GEMM and ALU architecture to implement depth-wise convolution. Detailed Implementation

[0021] To better understand the purpose, structure, and function of this invention, the following detailed description, in conjunction with the accompanying drawings, provides a method and apparatus for implementing depth-wise convolution using a fully reusable GEMM and ALU architecture.

[0022] This application provides a method and apparatus for implementing depth-wise convolution by fully reusing the GEMM and ALU architectures. For mobile and edge devices, it accelerates depth-wise convolution calculations without adding additional accelerators, while supporting standard convolution calculation acceleration. This improves the utilization of GEMM computing resources during depth-wise convolution operations, thereby increasing data throughput and processing performance. The main solution of this application is as follows: determining the GEMM and ALU architectures, determining the number and arrangement of multiply-accumulate units in the GEMM architecture, and determining the number of multiply-accumulate units in the ALU architecture; acquiring the input feature map data and weight data of the convolutional layer, dividing the input feature map data and weight data into several data blocks and sub-data blocks, and obtaining the calculation order of each data block; inputting the data from different channels of each input feature map data block and each weight sub-data block into the corresponding single-channel computing unit according to the calculation order; each single-channel computing unit performing convolution calculations on each input feature map data matrix and each weight sub-data matrix to obtain the output feature map data block; and writing the output feature map data block back to the corresponding address in memory in sequence.

[0023] The content of this application will be further explained below with reference to the accompanying drawings and through the description of the embodiments of this application.

[0024] This application provides a method for calculating depth-wise convolutions that fully reuses the GEMM and ALU architectures, such as... Figure 1 As shown, the method includes: Step 1: Determine the GEMM and ALU architectures, determine the number and arrangement of multiply-accumulate units in the GEMM architecture, and determine the number of multiply-accumulate units in the ALU architecture.

[0025] Specifically, the number of channels in the input feature map of the standard convolutional layer, the number of convolutional kernels, and the requirements of the application scenario are analyzed to determine the preset number of channels for parallel processing and the preset number of convolutional kernels for parallel processing. Based on the preset number of channels for parallel processing, the number of single-channel computation units in the GEMM architecture is determined. Based on the preset number of convolutional kernels for parallel processing, the number of multiply-accumulate units in each single-channel computation unit of the GEMM architecture and the number of multiply-accumulate units in the ALU architecture are determined. Since the Depth-wise convolutional layer performs channel-by-channel convolution, the computation results of each channel do not need to be accumulated. Therefore, the single-channel computation units of GEMM can be reused as input channels for Depth-wise convolution in parallel processing. Simultaneously, the multiply-accumulate units in each single-channel computation unit are reused as convolution computation units for the input feature map data and weight data in each channel of the Depth-wise convolution. In one implementation, the number of single-channel computation units in the GEMM architecture is equal to 8, and the number of multiply-accumulate units in each single-channel computation unit is equal to 8. In practical applications, the preset number of single-channel computation units and the preset number of multiply-accumulate units in each single-channel computation unit can be set according to application requirements, such as 16, 32, etc. For the standard convolutional layer using GEMM, the acceleration method is to process 8 convolutional kernels in parallel. Each convolutional kernel performs convolution calculations in parallel according to 8 channels. GEMM outputs the calculation results according to the output channel in each calculation cycle, resulting in a total of 8 output feature map data. For the depth-wise convolutional layer that reuses the GEMM architecture, the acceleration method is to process 8 channels in parallel. The 8 multiply-accumulators of each single-channel calculation unit perform convolution calculations on the 8 input feature map data and 8 weight data of each channel. GEMM outputs the calculation results according to the output channel in each calculation cycle, resulting in a total of 8 output feature map data.

[0026] Step 2: Obtain the input feature map data and weight data of the convolutional layer, divide the input feature map data and weight data into several data blocks and sub-data blocks, and obtain the calculation order of each data block.

[0027] Specifically, the input feature map data of the standard convolutional layer and the depth-wise convolutional layer are divided by channel dimension. The number of divisions is a preset number of channels for parallel processing, resulting in input feature map data blocks. Each input feature map data block has the same channel dimension, equal to the preset number of channels for parallel processing. The weight data of the standard convolutional layer and the depth-wise convolutional layer are also divided by channel dimension, with the number of divisions being a preset number of channels for parallel processing, resulting in weight data blocks. Each weight data block has the same channel dimension, equal to the preset number of channels for parallel processing. If the channel dimension of the input feature map data and the weight data is less than an integer multiple of the preset number, the number of channels in the input feature map and the weight data are adjusted to an integer multiple of the preset number by padding with zeros. Then, the adjusted input feature map data and the weight data are divided into data blocks using the same steps. According to the division method, the position index of the input feature map data block in the input feature map and the position index of the weight data block in the convolutional kernel are determined respectively.

[0028] The weight data blocks of the depth-wise convolutional layer are further divided into sub-weight data blocks of a preset size in the row and column dimensions. The preset size is determined based on the number of multiply-accumulate units in each single-channel computation unit of the GEMM. When a weight data block is less than an integer multiple of the preset size, its row and column dimensions are adjusted to an integer multiple of the preset size by padding with zeros in the rows and columns. Then, the adjusted weight data block is divided into sub-weight data blocks using the same steps. In one implementation, such as... Figure 2 As shown, the single-channel computation unit of the GEMM architecture uses 8 multiply-accumulate units, therefore the preset size is determined to be 3*3. In practical applications, the preset size can be set according to application requirements, such as 4*4, 6*6, etc., to adapt to the number of multiply-accumulate units in the single-channel computation unit. According to the division method of the weight data blocks, the position index of each sub-weight data block within the weight data block is obtained. The position index of each sub-weight data block within the weight data block is combined with the position index of each weight data block within the convolution kernel to obtain the position index of each sub-weight data block within the convolution kernel.

[0029] Based on the convolution calculation loop of standard convolutional layers and depth-wise convolutional layers, as well as the position index of each input feature map data block in the input feature map and the position index of each sub-weight data block in the convolution kernel, the convolution calculation order of the input feature map data blocks and each sub-weight data block of each convolutional layer is determined.

[0030] Step 3: According to the calculation order, input the data from different channels of each input feature map data block and each weight sub-data block into the corresponding single-channel calculation unit; each single-channel calculation unit performs convolution calculation on each input feature map data matrix and each weight sub-data matrix to obtain the output feature map data block.

[0031] Specifically, GEMM sequentially performs convolution calculations on the input feature map data blocks and sub-weight data blocks of each computation cycle to obtain the corresponding output feature map data blocks. Standard convolution in GEMM employs a parallel approach combining channel parallelism and kernel parallelism. Specifically, the single-channel computation unit of GEMM implements kernel parallelism for standard convolution, and the multiply-accumulate units in each single-channel computation unit implement channel parallelism. Depth-wise convolution in GEMM adopts a channel parallelism approach, with each channel performing independent Depth-wise convolution calculations. The single-channel computation unit of GEMM implements channel parallelism for Depth-wise convolution, and the multiply-accumulate units in each single-channel computation unit obtain independent convolution sums for each channel. If the size of the sub-weight data block of the Depth-wise convolution layer is larger than the number of multiply-accumulate units in a single-channel computation unit, when reusing GEMM to implement Depth-wise convolution, it is necessary to split the sub-weight data blocks and perform the convolution calculation of the input feature map data and weight data in two computation cycles. In one implementation, such as... Figure 3 As shown, the size of the sub-weight data block is 3*3, and the number of multiply-accumulate units in a single-channel computation unit is equal to 8. Therefore, the sub-weight data block needs to be split into two parts according to the number of multiply-accumulate units. The first part is the first 8 sub-weight data blocks, and the second part is the last sub-weight data block. In the first part, the weight data of each channel and the corresponding 8 input feature map data are input into the GEMM architecture channel by channel and the convolution calculation is completed. The calculation result is stored in a register specifically used to store the output result. In the second part, the weight data and the corresponding input feature map data are input into the multiply-accumulate units in the ALU architecture in channel order in the next cycle, and are simultaneously accumulated with the data in the register to obtain the convolution result of the sub-weight data block and the input feature map data block. In the convolution calculation of the first part, the utilization rate of the GEMM computation unit is 100%, and in the convolution calculation of the second part, the utilization rate of the ALU computation unit is 100%. During the second part of ALU computation, the GEMM architecture simultaneously performs convolution computation on the sub-weight data block and the first part of the next set of input feature map data. In each computation cycle, the GEMM's computational units are fully utilized. (The first part's weight data and the corresponding 8 input feature map data complete convolution computation in the first computation cycle to obtain an intermediate result. In the second computation cycle, the second part's weight data and the corresponding 1 input feature map data complete convolution computation, and this result is added to the intermediate result from the first computation cycle to obtain the output feature map data result from the convolution computation of a sub-weight data block and its corresponding input feature map data. In the first part of the convolution computation, the GEMM's computational units are fully utilized. In the second part of the convolution computation, some computational units in the GEMM are idle, but overall, it is better than a scheme where most computational units in the GEMM are idle in each computation cycle.) When processing standard convolutional layers, the input of the GEMM architecture in this application consists of input feature map data blocks and weight data blocks. The data in each weight data block comes from convolutional kernels processed in parallel, and the number of convolutional kernels processed in parallel is equal to the preset number of single-channel computation units. The number of channels in each input feature map data block and sub-weight data block processed by GEMM in each iteration is equal to the number of multiply-accumulate units in a single-channel computation unit. When processing depth-wise convolutional layers, the input of the GEMM and ALU architecture in this application consists of input feature map data blocks and sub-weight data blocks. The number of channels in each input feature map data block and each sub-weight data block is equal to the number of single-channel computation units. The number of input feature map data and weight data processed by each single-channel computation unit in each computation cycle of GEMM is less than or equal to the number of multiply-accumulate units in a single-channel computation unit.

[0032] The GEMM and ALU architecture in this application employs a four-layer loop to complete the depth-wise convolution. During the depth-wise convolution process, the first layer divides the input feature map into a predetermined number of groups (i.e., the number of input feature map data blocks) in parallel channel-wise manner, and the output feature map data is also output in parallel channel-wise manner. The second loop divides the weight data blocks into groups of a predetermined size (i.e., the number of sub-weight data blocks in each weight data block). Therefore, for the next three loops, this application uses weight reuse to implement depth-wise convolution, with each sub-weight data block corresponding to a partial sum of an output feature map data block. The third and fourth layers represent the length and width of the output feature map data block, respectively. GEMM obtains intermediate results of the output feature map row by row and writes them into on-chip memory specifically for storing these intermediate results. Simultaneously with the calculation of the output feature map result, an intermediate result at the same position as the current output feature map data is read from the on-chip buffer, added to the convolution sum obtained in the current calculation cycle, and then written back to the on-chip memory at the same address as when it was read. If the size of the sub-weight data block in the depth-wise convolutional layer is larger than the number of multiply-accumulate units in a single-channel computation unit, the sub-weight data block is split into two parts. The first part contains weight data equal to the number of multiply-accumulate units, and the second part contains the remaining weight data. Convolution calculations of the input feature map data and weight data are performed on both the GEMM and ALU architectures. When the second layer loop reaches its last iteration, the sum of these results is the final output feature map data.

[0033] Step 4: Write the output feature map data blocks back to the corresponding addresses in memory according to the calculation order.

[0034] Specifically, according to the description of the four-layer loop, the output feature data block is obtained after the last three loops. The size of this output feature map data block is equal to the parameters in the third and fourth loops. There are two cases for calculating the output feature map data: the first case is that the size of the weight data block is less than or equal to the preset size, and the second case is that the size of the weight data block is greater than the preset size.

[0035] If the size of the weighted data block is less than or equal to the preset size, there is only one weighted sub-data block, and the second loop will only be executed once. After the third and fourth loops are completed, the result is the final output feature map data block. Based on the position index of the current output data block in the entire output feature map data and the base address of the output data in the on-chip buffer, the starting address of the output feature map data block in the on-chip buffer is determined, and then the output feature map data block is written to the on-chip buffer.

[0036] If the size of the weight data block is larger than the preset size, there will be multiple weight sub-data blocks. The second loop will be executed multiple times. The inner two loops, i.e., the third and fourth loops, obtain intermediate results of the output feature map. This data is written into a register and participates in the accumulation in subsequent loops. After the second loop ends, all intermediate results with the same index are accumulated to obtain the final output feature map data. The number of channels in the output feature map of the Depth-wise convolutional layer is the same as the number of channels in the input feature map. Therefore, the index of the input channel number in the first loop is also the index of the output channel number. According to the index of the current output feature map data block and the base address of the output data in the on-chip buffer, the starting address of the output feature map data block in the on-chip buffer is determined, and the output feature map data block is written back to the on-chip buffer. After the first loop ends, all output feature map data is obtained and written sequentially to the off-chip memory according to the storage format of the output feature map data in the off-chip memory.

[0037] This application provides a computational device that fully reuses the GEMM and ALU architecture to implement depth-wise convolution, such as... Figure 4 As shown, the device includes: Off-chip memory and on-chip cache related to data storage, data transport module and depth-wise computing module related to data computation, wherein the depth-wise computing module includes GEMM architecture and ALU architecture.

[0038] The method for implementing depth-wise convolution by fully reusing the GEMM and ALU architecture determines the four-layer loop sequence of depth-wise convolution. The first layer is the number of groups into which the input feature map is divided according to a preset number of channels in parallel, i.e., the number of input feature map data blocks. The output feature map data is also output in a channel-parallel manner. The second layer is the number of groups into which the weight data blocks are divided according to a preset size, i.e., the number of sub-weight data blocks in each weight data block. Therefore, for the next three layers, this application adopts a weight reuse method to implement depth-wise convolution, with each sub-weight data block corresponding to a partial sum of an output feature map data block. The third and fourth layers are the length and width of the output feature map data blocks, respectively. The GEMM obtains the intermediate results of the output feature map row by row and writes them into on-chip memory specifically for storing the intermediate results of the output feature map. Based on these four layers, the transport order and calculation order of the input image data and weight data are determined. According to this transport order, the data transport module transports the input image data and weight data from off-chip memory to on-chip buffer.

[0039] The computation module includes a selector controlled by the computation mode field in the instruction. This field distinguishes between standard convolution and depth-wise convolution. For standard convolution, GEMM and ALU computations are performed independently in channel-parallel mode, determined by the computation type field in the instruction, which also determines the data to be read from the on-chip buffer during computation. For depth-wise convolution, GEMM and ALU computations are performed simultaneously, both using channel-parallel mode. Intermediate results for corresponding channels in the GEMM and ALU architectures are accumulated in the ALU to obtain the final output of the number of parallel channels. Based on the computation order of depth-wise convolution, the computation module determines the data address in the on-chip buffer and reads it into the GEMM and ALU architectures via the data interface. Intermediate results are calculated and accumulated in the GEMM architecture, serving as input data for the ALU architecture. This intermediate result is then added to the convolution results in the ALU to obtain the convolution sum of the number of parallel channels, which is written to the on-chip buffer. The device includes a selector controlled by the mode field in the instruction. Under the control of the first loop, the output results in the on-chip buffer are moved to the off-chip memory through the data transfer module, and the next round of Depth-wise convolution calculation of input image data and weight data begins.

[0040] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A computational method for depth-wise convolution that fully reuses the GEMM and ALU architecture, characterized in that, The method includes: Step 1: Determine the GEMM and ALU architectures, determine the number and arrangement of multiply-accumulate units in the GEMM architecture, and determine the number of multiply-accumulate units in the ALU architecture; Step 2: Obtain the input feature map data and weight data of the convolutional layer, divide the input feature map data and weight data into several data blocks and sub-data blocks, and obtain the calculation order of each data block; Step 2 includes the following specific steps: The input feature map data of the standard convolutional layer and the depth-wise convolutional layer are divided into channels by a predetermined number of channels for parallel processing. This results in input feature map data blocks, each with the same channel dimension, equal to the predetermined number of channels for parallel processing. The weight data of the standard convolutional layer and the depth-wise convolutional layer are also divided into channels by a predetermined number of channels for parallel processing. This results in weight data blocks, each with the same channel dimension, equal to the predetermined number of channels for parallel processing. If the channel dimension of the input feature map data and the weight data is less than an integer multiple of the predetermined number, the number of channels in the input feature map and the weight data is adjusted to an integer multiple of the predetermined number by padding the channels with zeros. Then, the adjusted input feature map data and the weight data are divided into data blocks using the same steps. Based on the division method, the position index of the input feature map data block in the input feature map and the position index of the weight data block in the convolutional kernel are determined respectively. The weight data blocks of the Depth-wise convolutional layer are further divided into sub-weight data blocks of a preset size in the row and column dimensions. The preset size is determined according to the number of multiply-accumulators in each single-channel computing unit of GEMM. When the weight data block is less than an integer multiple of the preset size, the row and column dimensions of the weight data block are adjusted to an integer multiple of the preset size by padding with zeros in the rows and columns. Then, the adjusted weight data block is divided into sub-weight data blocks in the same way. Based on the convolution calculation loop of the standard convolutional layer and the depth-wise convolutional layer, as well as the position index of each input feature map data block in the input feature map and the position index of each sub-weight data block in the convolutional kernel, the convolution calculation order of the input feature map data blocks and each sub-weight data block of each convolutional layer is determined. Step 3: According to the calculation order, input the data from different channels of each input feature map data block and each weight sub-data block into the corresponding single-channel calculation unit; each single-channel calculation unit performs convolution calculation on each input feature map data matrix and each weight sub-data matrix to obtain the output feature map data block; Step 4: Write the output feature map data blocks back to the corresponding addresses in memory according to the calculation order.

2. The method for calculating depth-wise convolution by fully reusing the GEMM and ALU architecture according to claim 1, characterized in that, Step 1 includes the following specific steps: Analyzing the number of channels in the input feature map of the standard convolutional layer, the number of convolutional kernels, and the requirements of the application scenario, the preset number of channels parallel and the preset number of convolutional kernels parallel are determined respectively. Based on the preset number of channels parallel, the number of single-channel computation units in the GEMM architecture is determined. Based on the preset number of convolutional kernels parallel, the number of multiply-accumulate units in each single-channel computation unit in the GEMM architecture and the number of multiply-accumulate units in the ALU architecture are determined. The single-channel computation units of GEMM are reused as input channels parallel for Depth-wise convolution. At the same time, the multiply-accumulate units in each single-channel computation unit are reused as convolution computation units for the input feature map data and weight data in each channel of Depth-wise convolution.

3. The method for calculating depth-wise convolution by fully reusing the GEMM and ALU architecture according to claim 1, characterized in that, In step 1, for the standard convolutional layer using GEMM, the acceleration method is to process 8 convolutional kernels in parallel. Each convolutional kernel performs convolution calculations in parallel according to 8 channels. GEMM outputs the calculation results according to the output channel in each calculation cycle, resulting in a total of 8 output feature map data. For the depth-wise convolutional layer that reuses the GEMM architecture, the acceleration method is to process 8 channels in parallel. The 8 multiply-accumulators of each single-channel calculation unit perform convolution calculations on the 8 input feature map data and 8 weight data of each channel. GEMM outputs the calculation results according to the output channel in each calculation cycle, resulting in a total of 8 output feature map data.

4. The method for calculating depth-wise convolution by fully reusing the GEMM and ALU architecture according to claim 1, characterized in that, Step 3 includes the following specific steps: GEMM sequentially performs convolution calculations on the input feature map data blocks and sub-weight data blocks of each computation cycle to obtain the corresponding output feature map data blocks. Standard convolution in GEMM adopts a parallel approach with both channel parallelism and kernel parallelism. Specifically, the single-channel computation unit of GEMM implements kernel parallelism for standard convolution, and the multiply-accumulate units in each single-channel computation unit implement channel parallelism. Depth-wise convolution in GEMM adopts a channel parallel approach, with each channel performing Depth-wise convolution calculations independently. The single-channel computation unit of GEMM implements channel parallelism for Depth-wise convolution, and the multiply-accumulate units in each single-channel computation unit obtain independent convolution sums for each channel. If the size of the sub-weight data block of the Depth-wise convolution layer is larger than the number of multiply-accumulate units in a single-channel computation unit, the sub-weight data block needs to be split when reusing GEMM to implement Depth-wise convolution, and the convolution calculation of the input feature map data and weight data needs to be completed in two computation cycles.

5. The method for calculating depth-wise convolution by fully reusing the GEMM and ALU architecture according to claim 1, characterized in that, In step 3, when processing standard convolutional layers, the input of the GEMM architecture is input feature map data blocks and weight data blocks. The data in each weight data block comes from convolutional kernels processed in parallel. The number of convolutional kernels processed in parallel is equal to the preset number of single-channel computation units. The number of channels in each input feature map data block and sub-weight data block processed by GEMM in each cycle is equal to the number of multiply-accumulate units in a single-channel computation unit. When processing depth-wise convolutional layers, the input of the GEMM and ALU architectures is input feature map data blocks and sub-weight data blocks. The number of channels in each input feature map data block and each sub-weight data block is equal to the number of single-channel computation units. The number of input feature map data and weight data processed by each single-channel computation unit in each computation cycle of GEMM is less than or equal to the number of multiply-accumulate units in a single-channel computation unit.

6. The method for calculating depth-wise convolution by fully reusing the GEMM and ALU architecture according to claim 1, characterized in that, In step 3, the GEMM and ALU architecture computes depth-wise convolution using a four-layer loop. During this process, the first layer divides the input feature map into a predetermined number of groups (i.e., the number of input feature map data blocks) in parallel channel-wise manner, and the output feature map data is also output in parallel channel-wise. The second loop divides the weight data blocks into groups of a predetermined size (i.e., the number of sub-weight data blocks within each weight data block). Therefore, for the next three loops, depth-wise convolution is implemented using weight reuse, with each sub-weight data block corresponding to a partial sum of an output feature map data block. The third and fourth layers define the length and width of the output feature map data blocks, respectively. GEMM obtains the intermediate results of the output feature map row by row and writes them to a dedicated storage unit. In the on-chip memory of the intermediate results of the feature map, while calculating the output feature map result, the intermediate result at the same position as the current output feature map data is read from the on-chip buffer, and added to the convolution sum obtained in the current calculation cycle. After obtaining the new intermediate result, it is written back to the on-chip memory at the same address as when it was read. If the size of the sub-weight data block of the depth-wise convolutional layer is greater than the number of multiply-accumulate units in a single-channel computing unit, the sub-weight data block is split into two parts. The number of weight data in the first part is equal to the number of multiply-accumulate units, and the second part is the remaining weight data. The convolution calculation of the input feature map data and weight data is completed on the GEMM architecture and ALU architecture, respectively. When the second layer loop is executed to the last one, the sum is the final data of the output feature map data.

7. The method for calculating depth-wise convolution by fully reusing the GEMM and ALU architecture according to claim 1, characterized in that, Step 4 includes the following specific steps: Based on the description of the four-layer loop in Step 3, after the last three loops, an output feature data block is obtained. The size of this output feature map data block is equal to the parameters in the third and fourth loops. There are two cases for calculating the output feature map data: the first case is that the size of the weight data block is less than or equal to a preset size; the second case is that the size of the weight data block is greater than a preset size. If the size of the weight data block is less than or equal to the preset size, there is only one weight sub-data block, and the second loop will only be executed once. After the third and fourth loops are completed, the result is the final output feature map data block. Based on the position index of the current output data block in the entire output feature map data and the base address of the output data in the on-chip buffer, the starting address of the output feature map data block in the on-chip buffer is determined, and then the output feature map data block is written into the on-chip buffer. If the size of the weight data block is greater than the preset size… If the output feature map is a single block of data, there will be multiple weighted sub-data blocks. The second loop will be executed multiple times. The inner two loops, namely the third and fourth loops, obtain intermediate results of the output feature map. This data is written into a register and participates in the accumulation in subsequent loops. After the second loop ends, all intermediate results with the same index are accumulated to obtain the final output feature map data. The number of channels in the output feature map of the depth-wise convolutional layer is the same as the number of channels in the input feature map. Therefore, the index of the input channel number in the first loop is also the index of the output channel number. According to the index of the current output feature map data block and the base address of the output data in the on-chip buffer, the starting address of the output feature map data block in the on-chip buffer is determined, and the output feature map data block is written back to the on-chip buffer. After the first loop ends, all output feature map data is obtained and written sequentially to the off-chip memory according to the storage format of the output feature map data in the off-chip memory.

8. A computing device that fully reuses the GEMM and ALU architecture to implement depth-wise convolution, characterized in that, include: Off-chip memory and on-chip cache related to data storage; data transport module and depth-wise computing module related to data computation, wherein the depth-wise computing module includes GEMM architecture and ALU architecture; The GEMM architecture and ALU architecture determine the four-layer loop order of the Depth-wise convolution. The first layer is the number of groups into which the input feature map is divided according to a preset number of channels in parallel, i.e., the number of input feature map data blocks. The output feature map data is also output in a channel-parallel manner. The second layer is the number of groups into which the weight data blocks are divided according to a preset size, i.e., the number of sub-weight data blocks in each weight data block. Therefore, for the next three layers, the weight reuse method is used to implement the Depth-wise convolution. Each sub-weight data block corresponds to a partial sum of an output feature map data block. The third and fourth layers are the length and width of the output feature map data block, respectively. GEMM obtains the intermediate results of the output feature map row by row and writes them into the on-chip memory that is dedicated to storing the intermediate results of the output feature map. Based on these four layers, the transport order and calculation order of the input image data and weight data are determined. According to this transport order, the data transport module transports the input image data and weight data from the off-chip memory to the on-chip buffer. The computation module includes a selector controlled by a computation mode field in the instructions. This field distinguishes between standard convolution and depth-wise convolution. For standard convolution, GEMM and ALU computations are performed independently in channel-parallel mode, determined by the computation type field in the instructions, which also determines the data to be read from the on-chip buffer during computation. For depth-wise convolution, GEMM and ALU computations are performed simultaneously, both using channel-parallel mode. Intermediate results for corresponding channels in the GEMM and ALU architectures are accumulated in the ALU to obtain the final output of the number of parallel channels. Based on the computation order of depth-wise convolution, the computation module determines the data address in the on-chip buffer and reads it into the GEMM and ALU architectures via a data interface. Intermediate results are calculated and accumulated in the GEMM architecture, serving as input data for the ALU architecture. This intermediate result is then added to the convolution results in the ALU to obtain the convolution sum of the number of parallel channels, which is written to the on-chip buffer. The device includes a selector controlled by a mode field in the instructions. Under the control of the first loop, the output results in the on-chip buffer are moved to the off-chip memory through the data transfer module, and the next round of Depth-wise convolution calculation of input image data and weight data begins.

Images