A three-dimensional convolution parallel computing method, device and equipment based on multi-core processor

Abstract

The present invention relates to the intersection of deep learning and high-performance computing, and in particular to a three-dimensional convolution parallel computing method, device and equipment based on a multi-core processor. The method includes: according to the available space size of the memory of the slave core array of the multi-core processor, the input feature map is divided into multiple sub-input feature maps in the direction parallel to its third dimension; the convolution kernel is obtained, and the sub-input feature map plane matrix equal to the number of convolution kernel plane matrices contained in the convolution kernel is taken from the sub-input feature map, and the plane matrix is ​​sent to the slave core array together with the convolution kernel to perform three-dimensional matrix convolution calculation; the three-dimensional matrix convolution calculation is divided into a number of computing units of equal number corresponding to the number of available slave cores in the slave core array, and each slave core is responsible for the calculation of a computing unit. During the calculation process of the computing unit, data exchange is performed in the memory of the slave core array, and the slave cores are calculated in parallel. After all slave cores have completed the calculation, the results are written back to the main memory.

Description

Technical Field

[0001] The present invention relates to the intersection of deep learning and high performance computing, and in particular to a three-dimensional convolution parallel computing method, device and equipment based on a multi-core processor. Background Art

[0002] Three-dimensional convolution is an important data processing technology in the fields of computer vision and deep learning, and is widely used in scientific computing for processing three-dimensional data, such as three-dimensional model analysis, video processing, medical imaging, fluid dynamics, molecular dynamics, etc.

[0003] In convolutional neural networks (CNNs), three-dimensional convolution operations can capture the characteristics of the spatial and temporal (or depth) dimensions of data, thereby extracting richer and more complex information, which is particularly critical in applications that combine high-performance computing (HPC) and artificial intelligence (AI).

[0004] The new generation of Sunway supercomputer is equipped with the latest SW26010Pro heterogeneous multi-core processor, which is one of the most advanced supercomputer platforms in my country and has important strategic significance in the international high-performance field. Many scientific computing and artificial intelligence applications are deployed on the Sunway platform, and three-dimensional convolution is a hot operator and a key factor affecting program performance. Therefore, it is very necessary to optimize the three-dimensional convolution for the SW26010Pro architecture.

[0005] However, although the SW26010Pro heterogeneous multi-core processor has very powerful computing and storage capabilities, the three-dimensional convolution performance on it is far from satisfactory. Currently, the algorithm optimization of three-dimensional convolution on the SW26010Pro architecture is still in its early stages, and most of them are for two-dimensional convolution, such as the swDNN library.

[0006] There are differences in computational complexity and storage structure between two-dimensional convolution and three-dimensional convolution. Compared with traditional two-dimensional convolution, three-dimensional convolution adds an additional time (or depth) dimension, which exponentially increases the size of input data, convolution kernel, and output data, resulting in high computational complexity and memory consumption.

[0007] The existing two-dimensional convolution algorithm on the Sunway supercomputer platform is not suitable for three-dimensional convolution with huge data volume and calculation amount. For example, the block algorithm of two-dimensional convolution using the swDNN library will cause a large amount of redundant data to be read from the main memory during the calculation of three-dimensional convolution, resulting in a decrease in the calculation access memory ratio. In addition, due to the large amount of memory access for three-dimensional convolution, without the use of merged memory access optimization, the main memory bandwidth is low, resulting in unacceptable time consumed for memory access, especially in the process of high-precision and large-size three-dimensional convolution calculation.

[0008] Therefore, it is necessary to propose a three-dimensional convolution calculation optimization method for multi-core processors to improve the computing performance of three-dimensional convolution on them. Summary of the invention

[0009] One or more embodiments of this specification describe a method, apparatus, and device for three-dimensional convolution parallel computing based on a multi-core processor. Based on the multi-core processor, the three-dimensional convolution computing is optimized by utilizing limited slave core storage space, and a block parallel algorithm is designed to improve the overall performance of the three-dimensional convolution computing.

[0010] According to a first aspect, a three-dimensional convolution parallel computing method based on a multi-core processor is provided, which is applied to a multi-core processor, wherein a core group of the multi-core processor includes a master core and a slave core array having multiple slave cores, and comprises the following steps:

[0011] Obtaining a three-dimensional convolution calculation task, including an input feature map; the input feature map has a plurality of planar matrices consisting of a first dimension and a second dimension arranged in a direction of a third dimension;

[0012] According to the available memory space size of the slave core array of the multi-core processor, the master core divides the input feature map into a plurality of sub-input feature maps in a direction parallel to the third dimension thereof, to obtain a three-dimensional convolution calculation subtask;

[0013] In response to the three-dimensional convolution calculation subtask, the main core obtains a convolution kernel, the convolution kernel is composed of a plurality of convolution kernel plane matrices, the main core takes a number of sub-input feature map plane matrices equal to the number of convolution kernel plane matrices contained in the convolution kernel from the sub-input feature map along the direction of the third dimension, and sends them together with the convolution kernel into the slave core array to perform three-dimensional matrix convolution calculation;

[0014] The three-dimensional matrix convolution calculation is divided into a number of computing units of equal number corresponding to the number of available slave cores in the slave core array, each of the slave cores is responsible for the calculation of one of the computing units, and during the calculation process of the computing units, data exchange is performed in the memory of the slave core array, and the slave cores are calculated in parallel. After all the slave cores have completed the calculation, the results are written back to the main memory.

[0015] According to one embodiment, the input feature map and the convolution kernel both have channel dimension parameters, and the method further includes:

[0016] The vector data of the input feature map and the convolution kernel are stored with the channel dimension as the lowest dimension.

[0017] According to one embodiment, the method further includes: preloading the data of the convolution kernel into the memory of the slave core array and retaining the data therein until the three-dimensional convolution calculation task is completed.

[0018] According to one embodiment, writing the result back to the main memory includes: rearranging the result data of the three-dimensional matrix convolution calculation obtained from the kernel array calculation in the same dimension, and writing the rearranged data back to the main memory.

[0019] In one embodiment of the above implementation, the above method further includes: using RMA communication to exchange data between the slave cores of the slave core array to complete the rearrangement of the result data of the three-dimensional matrix convolution calculation in the same dimension.

[0020] According to one embodiment, each of the slave cores is responsible for the calculation of one of the computing units, including: the slave core uses implicit matrix multiplication to calculate the computing unit.

[0021] According to one embodiment, the method further comprises:

[0022] Iteratively performing the three-dimensional matrix convolution calculation along the third dimension direction of the sub-input feature map until all plane matrices of the sub-input feature map are calculated;

[0023] When the first iterative calculation is completed, within the data range required for the second iterative calculation, part of the loaded data continues to reside in the memory of the slave core array, and the remaining unread data is loaded into the memory of the slave core array.

[0024] In one embodiment of the above implementation manner, the remaining unread data is loaded into the memory of the slave core array, including: loading the remaining unread data through an asynchronous DMA (Direct Memory Access) mode.

[0025] According to the second aspect, the present invention also provides a three-dimensional convolution parallel computing device based on a multi-core processor, the device is deployed on a platform with a multi-core processor, a core group of the multi-core processor includes a master core and a slave core array with multiple slave cores, specifically including:

[0026] An acquisition module: configured to acquire a three-dimensional convolution calculation task, comprising an input feature map; the input feature map has a plurality of plane matrices consisting of a first dimension and a second dimension arranged in a direction of a third dimension;

[0027] A segmentation module: configured to divide the input feature map into a plurality of sub-input feature maps in a direction parallel to the third dimension thereof according to the available memory space size of the slave core array of the multi-core processor, so as to obtain a three-dimensional convolution calculation subtask;

[0028] A three-dimensional matrix convolution preparation module: configured to respond to the three-dimensional convolution calculation subtask, the main core obtains a convolution kernel, the convolution kernel is composed of a plurality of convolution kernel plane matrices, the main core takes a number of sub-input feature map plane matrices equal to the number of convolution kernel plane matrices contained in the convolution kernel from the sub-input feature map along the direction of the third dimension, and sends them together with the convolution kernel into the slave core array to perform three-dimensional matrix convolution calculation;

[0029] Three-dimensional matrix convolution calculation module: configured to divide the three-dimensional matrix convolution calculation into a number of equal number of calculation units corresponding to the number of available slave cores in the slave core array, each of the slave cores is responsible for the calculation of one of the calculation units, and during the calculation process of the calculation unit, data exchange is performed in the memory of the slave core array, and the slave cores are calculated in parallel. After all the slave cores have completed the calculation, the results are written back to the main memory.

[0030] According to a third aspect, the present invention further provides a computing device, comprising a processor and a memory, wherein the processor is configured to execute instructions stored in the memory to execute the method described in the first aspect.

[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0032] First, the present invention converts the original single-level calculation with high complexity into a more efficient multi-level operation by designing a multi-level block calculation mode of three-dimensional convolution, which can reduce the complexity of each calculation step and make it more suitable for parallel processing, thereby significantly improving the calculation efficiency of three-dimensional convolution;

[0033] Secondly, the present invention realizes the reuse of convolution kernel data and input feature map data by designing a convolution kernel data retention mechanism in the memory of the slave kernel array and designing an iterative calculation mechanism of three-dimensional matrix convolution, thereby greatly reducing the reading of redundant data;

[0034] Thirdly, the present invention achieves the overlap of calculation and memory access by using the asynchronous DMA mode to load the data required for calculation, thus maximally reducing the time for calculation to wait for data;

[0035] Finally, the present invention fully utilizes the hardware resources of the multi-core processor by dividing the matrix convolution calculation task into slave cores for parallel calculation, thereby improving the overall performance of three-dimensional convolution calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a flow chart of a three-dimensional convolution parallel computing method based on a multi-core processor provided in an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the technical architecture of a three-dimensional convolution parallel computing method based on a multi-core processor provided in an embodiment of the present invention;

[0038] Figure 3 A schematic diagram of a multi-level buffering technology in a three-dimensional convolution parallel computing method based on a multi-core processor provided in an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a three-dimensional convolution parallel computing device based on a multi-core processor provided in an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of a three-dimensional convolution parallel computing device based on a multi-core processor provided in an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of an input feature map storage structure provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention proposes a three-dimensional convolution parallel computing method, device and equipment based on a multi-core processor. In order to enable technicians in this technical field to better understand the scheme of the present invention, the technical scheme of the present invention is further described in detail through the accompanying drawings and embodiments.

[0043] In a multi-core processor, there are multiple layers of memory / cache corresponding to different core levels. The lower the cache level, the closer it is to the processor core, which means faster storage speed, but at the same time, the smaller its storage space; therefore, it is necessary to consider how to maximize the use of cache storage space and make targeted optimizations while optimizing the algorithm.

[0044] For example, in the storage architecture of the domestic Shenwei processor, the memory can be divided into three levels, from high to low, namely: global memory (Global), local data memory (LDM) and register (Register). From high to low, the storage capacity gradually decreases and the memory access speed gradually increases.

[0045] In order to utilize limited LDM and register resources to achieve efficient computing performance, the present invention designs a three-level blocking algorithm for three-dimensional convolution calculation, which divides large-scale computing tasks into smaller computing subtasks and computing units, and calculates them in sequence, thereby making full use of the storage space of the cache, improving the memory access ratio, and realizing efficient three-dimensional convolution calculation.

[0046] The following describes in detail the specific implementation process of a three-dimensional convolution parallel computing method based on a multi-core processor provided by the present invention under the above concept.

[0047] The method is applied to a multi-core processor, wherein a core group in the multi-core processor includes a master core and a slave core array having a plurality of slave cores.

[0048] In one example, the method is implemented based on a Sunway SW26010Pro processor, in which a core group has a master core and a slave core array including 64 slave cores, and the 64 slave cores are arranged in a matrix of 8 rows and 8 columns.

[0049] Both the master core and the slave core can access the main memory, and the slave core can access the main memory through DMA (Direct Memory Access).

[0050] Each slave core contains a local data memory (LDM), which can store the programs running on the slave core and the data required for the slave core to run. Each slave core can read and write its LDM at high speed. The slave cores share data through RMA communication. The RMA bandwidth is much larger than the DMA bandwidth.

[0051] It is understandable that, in other examples, the multi-core processor may also be other types of processors with similar hardware structures, which is not specifically limited here.

[0052] Figure 1 FIG. 2 shows a schematic diagram of a three-dimensional convolution parallel computing method based on a multi-core processor according to an exemplary embodiment of the present invention. Figure 1 , the calculation method specifically includes the following steps:

[0053] S101: Obtain a three-dimensional convolution calculation task, including an input feature map; the input feature map has a plurality of planar matrices composed of a first dimension and a second dimension arranged in a direction of a third dimension.

[0054] The three-dimensional convolution calculation task includes a three-dimensional spatial input feature map composed of three dimensions; and also has a three-dimensional spatial convolution kernel composed of the same three dimensions as the input feature map.

[0055] In another embodiment, the input feature map may also have a channel dimension, which represents the number of channels possessed by the input feature map; the convolution kernel has the same channel dimension and number of channels as the input feature map. It is understandable that when the input feature map is multi-channel, the three-dimensional convolution calculation task is equivalent to performing calculations on each channel separately with the same three-dimensional convolution calculation logic, obtaining multiple output feature maps with the same number of channels, and then adding the corresponding positions of the output feature maps to obtain the calculation result of the three-dimensional convolution calculation task. Therefore, the multi-channel input feature map performs a three-dimensional convolution calculation with a convolution kernel with the same number of channels, and a single-channel output feature map will be obtained.

[0056] In another embodiment, the convolution kernel may also have an output channel dimension, representing the number of output results expected to be obtained by the three-dimensional convolution calculation task. It is understandable that when the convolution kernel has multiple output channels, it is equivalent to the three-dimensional convolution calculation task having multiple convolution kernels, and the input feature map is subjected to three-dimensional convolution calculation with each convolution kernel to obtain multiple output feature maps with the same number of output channels, which is the calculation result of the three-dimensional convolution calculation task.

[0057] In one example, for example Figure 2 As shown, the input feature map 201 is a three-dimensional spatial feature map composed of three dimensions of width W (Width), height H (Height) and depth D (Depth), which has a channel C (Channel) attribute; the convolution kernel 202 has the same dimensional attributes as the input feature map, and also has an output channel C o property.

[0058] In other examples, according to different three-dimensional convolution computing tasks, the input feature map and the convolution kernel may have different dimensional definitions and dimensional parameters, which are not specifically limited here.

[0059] In another embodiment, taking into account the subsequent convolution calculation, implicit matrix multiplication and vectorized computing components can be used to accelerate multi-channel vector calculations. In one vectorized accelerated calculation cycle, it is necessary to read the data of all channels at the same spatial position at one time for calculation; therefore, when storing the input feature map and the convolution kernel, the data structure can also be adjusted in the storage dimension, using the channel dimension as the lowest dimension of the storage data structure to store the vector data of the input feature map and the convolution kernel.

[0060] In an example of the above embodiment, in a three-dimensional convolution calculation task, D, H, W, and C are used to represent the four dimensions of depth, height, width, and channel, respectively, and the channel dimension is used as the lowest dimension for data storage. Then the data structure of the input feature map can be expressed as (D i , H i , W i , C i ), similarly, the data structure of the convolution kernel can be expressed as (k_d, k_h, k_w, C i , C o ), where C o is the output channel of the convolution kernel. Therefore, we can get the output target of the three-dimensional convolution calculation task as (D o , H o , W o , C o ).

[0061] For example, the depth, height, width and number of channels of the input feature map are all 3. Figure 6 The input feature map storage structure diagram is shown; Figure 6 As shown, the data is stored in a way that the channel dimension is the lowest dimension of data storage, and the elements at the same spatial position in all channels in the memory are stored sequentially, followed by the elements from the next spatial position. It can be understood that Figure 6 It is only a schematic diagram of the relative position relationship of data elements in the memory, and does not mean that the row and column addresses of the memory in the embodiment are arranged as shown in the figure.

[0062] Storing in this way, there is no need to consider the depth, height, and width of the input feature map. During the calculation process, the processor can ensure that all channel data at the required specific spatial location is read continuously at one time, thereby preparing data access for the use of vectorized computing components to accelerate calculations in subsequent steps.

[0063] S103: According to the available memory space size of the slave core array of the multi-core processor, the master core divides the input feature map into multiple sub-input feature maps in a direction parallel to the third dimension thereof to obtain a three-dimensional convolution calculation subtask.

[0064] In the subsequent steps of slave kernel array calculation, the slave kernel array will be responsible for the end-to-end convolution operation in the third dimension of the input feature map. The end-to-end convolution operation means that within a slave kernel array operation cycle, the slave kernel array aims to completely output a plane consisting of the first dimension and the second dimension of the input feature map, and performs a three-dimensional matrix convolution operation on the plane by sliding the convolution kernel.

[0065] After one operation cycle is completed, the convolution kernel is slid one unit along the third dimension of the input feature map to enter the next operation cycle, and the calculation is repeated in this way until all planes of the input feature map are calculated.

[0066] As mentioned above, the access speed of the slave core array to the LDM is much higher than its access speed to the main memory. Therefore, in order to improve the overall computing efficiency, in a unit operation cycle of the slave core array, the calculated process data will be temporarily stored in the memory of the slave core array, and the results will be written back to the main memory after all calculations of the operation cycle are completed.

[0067] If the amount of planar data composed of the first dimension and the second dimension in the input feature map is too large, the process data that the LDM needs to temporarily store will exceed the storage capacity of the LDM within a unit operation cycle of the slave core array, and data will have to be exchanged with the main memory in the middle of the operation, thereby affecting the memory access speed and reducing the computing performance of the slave core array.

[0068] In one embodiment, according to the available space size of the LDM memory of the slave core array of the multi-core processor, the input feature map is divided into multiple sub-input feature maps in a direction parallel to its third dimension to obtain multiple three-dimensional convolution calculation subtasks, and each three-dimensional convolution calculation subtask is a calculation block.

[0069] For example Figure 2 As shown, the calculation block 203 is a calculation subtask split from the input feature map 201 in a direction parallel to the depth dimension. In the unit operation cycle of the kernel array, the output plane matrix 204 will be calculated; this is because in the unit operation cycle, the input feature map and the convolution kernel have the same depth parameter k and the same number of channels C. i , therefore, the output feature map is a two-dimensional plane matrix with an output width bW o and output height bH o , after the calculation is completed, the result is written back to the main memory.

[0070] By dividing the computing subtasks, the amount of data that the slave core array needs to temporarily store in a unit computing cycle is reduced, thereby reducing data exchange with the main memory during the computing process and ensuring the computing speed of the slave core array.

[0071] Based on the above technical concept, in the description of the next step, in combination with the embodiment, the calculation method of a calculation block will be explained in detail.

[0072] S105: In response to the three-dimensional convolution calculation subtask, the main core obtains a convolution kernel, which is composed of multiple convolution kernel plane matrices. The main core takes sub-input feature map plane matrices along the direction of the third dimension from the sub-input feature map, which are equal to the number of convolution kernel plane matrices contained in the convolution kernel, and sends them together with the convolution kernel to the slave core array to perform three-dimensional matrix convolution calculation.

[0073] In one embodiment, the third dimension is set as the depth dimension, and the depth parameter of the convolution kernel is recorded as k. The goal of a single three-dimensional matrix convolution calculation is to obtain an output plane matrix by performing a three-dimensional matrix convolution calculation on the k plane matrices of the sub-input feature map and the convolution kernel; the three-dimensional matrix convolution calculation is iteratively performed along the third dimension direction of the sub-input feature map until all plane matrices of the sub-input feature map in the third dimension are traversed and calculated, and a sub-output feature map containing multiple output plane matrices is obtained.

[0074] In an example of the above embodiment, taking one of the three-dimensional matrix convolution calculations as an example, the output plane matrix is ​​recorded as the m-th plane matrix out_D[m] of the sub-output feature map, the m-th plane matrix of the sub-input feature map is recorded as in_D[m], and the n-th plane matrix of the convolution kernel is recorded as filter_D[n]. Assuming that the depth parameter k of the convolution kernel is 3, the formula for a single three-dimensional matrix convolution calculation can be expressed as:

[0075]

[0076] It can be seen that in the iterative process of the three-dimensional matrix convolution calculation, part of the input data required for the previous calculation can be used as the input of the current calculation. Therefore, in the iterative calculation process, multi-level data buffering technology can be used to store reusable data in the cache, thereby improving data reuse rate and reducing the reading of redundant data.

[0077] In one embodiment, the three-dimensional matrix convolution calculation is iteratively performed along the third dimension of the sub-input feature map, including: after the first iterative calculation is completed, within the data range required for the second iterative calculation, part of the loaded data continues to reside in the LDM, and the remaining unread data is loaded into the LDM.

[0078] In one example of the above embodiment, Figure 3 According to this exemplary embodiment, a multi-level buffering technology provided by the present invention is shown. Figure 3: Assume that the depth parameter of the convolution kernel is k=3; in the first iterative calculation, the output is out_D[m], and the required plane matrices of the input feature map are in_D[m-1], in_D[m], and in_D[m+1]; the second iterative calculation output is out_D[m+1], and the required plane matrices of the input feature map are in_D[m], in_D[m+1], and in_D[m+2]; and so on.

[0079] exist Figure 3 In the figure, the input data involved in the current three-dimensional matrix convolution calculation is marked as [a]. It can be seen that when the first iteration is performed, the required three plane matrix data are all loaded into LDM; after the first iteration calculation is completed, the plane matrix data in_D[m] and in_D[m+1] required for the second iteration continue to reside in LDM, the cache space of in_D[m-1] is released, and the data in_D[m+2] required for the second iteration that has not been loaded yet is read.

[0080] According to the above example, compared with the calculation method that does not use the data multi-level buffering technology, in one iterative calculation process, the data reading amount is reduced from 3 plane matrix data to 1 plane matrix data, which greatly reduces the reading of redundant data.

[0081] In another example of the above embodiment, in order to avoid the problem of blocking the iterative calculation due to data reading, during the first iterative calculation, the data required for the second iteration but not yet read is loaded through the asynchronous DMA mode. Figure 3 In the figure, the cache space marked as [in] shows that during the calculation of out_D[m] in the first iteration, the in_D[m+2] data required for the second iteration is asynchronously loaded; during the calculation of out_D[m+1] in the second iteration, the in_D[m+3] data required for the next iteration is asynchronously loaded; and so on. Asynchronous DMA allows data to be loaded in the background without blocking the calculation tasks of the master and slave cores, which means that the slave core can access memory while performing calculations, achieving overlapping calculations and memory accesses, and avoiding the problem of blocking iterative calculations due to data reading.

[0082] In another embodiment, during the processing of the three-dimensional convolution calculation task, the convolution kernel data is preloaded into the LDM and resides there until the three-dimensional convolution calculation task is completed.

[0083] In three-dimensional convolution calculations, the convolution kernel data will be frequently used as input parameters in each convolution calculation. Each use means reading the convolution kernel data from the main memory. The convolution kernel data will not change during the entire three-dimensional convolution calculation process, which will generate a large number of memory access operations between the main memory and LDM and redundant data reading.

[0084] Before performing three-dimensional convolution calculations, the convolution kernel data is preloaded into the LDM and remains there until the calculation is completed. Each convolution calculation can directly use the convolution kernel data residing in the LDM without frequently accessing the main memory, minimizing the reading and memory access operations of redundant data, and further improving the performance of three-dimensional convolution calculations.

[0085] On the basis of the above method, in order to further accelerate the three-dimensional matrix convolution calculation, multiple slave cores of the processor slave core array are applied to the three-dimensional matrix convolution calculation to realize parallel calculation and speed up the calculation speed. The method is described in detail through an embodiment below.

[0086] S107: Divide the three-dimensional matrix convolution calculation into a number of computing units of equal number corresponding to the number of available slave cores in the slave core array, each of the slave cores is responsible for the calculation of one of the computing units, and during the calculation process of the computing units, data exchange is performed in the memory of the slave core array, and the slave cores perform calculations in parallel. After all the slave cores have completed the calculations, the results are written back to the main memory.

[0087] In one example, the slave core array in a core group of the Shenwei SW26010Pro processor has 64 slave cores, and the 64 slave cores are arranged in a matrix of 8 rows and 8 columns. Figure 2 As shown, a three-dimensional matrix convolution calculation is evenly divided into 64 computing units, which are distributed to 64 slave cores 205, and the division dimensions are height and output channel dimensions; it can be seen that the computing unit output on a single slave core is (1, bH o / 8,bW o , C o / 8), the slave cores perform parallel calculations. When all the slave cores have completed the calculations, that is, when the plane matrix required for the three-dimensional matrix convolution calculation is calculated, the result is written back to the main memory. It can be understood that in other examples, the slave core array of the multi-core processor can have other numbers of slave cores and different slave core arrangements, and the division dimension of the computing unit can also be other dimensions.

[0088] In one embodiment, an asynchronous DMA mode may be used to write the calculation results of the slave core array back to the main memory without blocking the calculation process, thereby achieving overlapping operations of calculation and memory access.

[0089] In another embodiment, implicit matrix multiplication (Implicit-GEMM) is used to perform convolution calculations in the computing unit, and the three-dimensional convolution kernel in the three-dimensional convolution calculation task is expanded into a two-dimensional matrix structure in the channel direction. The input feature map is also expanded and mapped into a two-dimensional matrix according to the expanded form of the convolution kernel. In this way, the matrix multiplication and addition calculation in the convolution calculation is converted into a matrix multiplication calculation with the channel dimension as the accumulation direction, making full use of the divisible and parallel calculation characteristics of matrix multiplication to accelerate the convolution calculation process. At the same time, implicit matrix multiplication will not explicitly rearrange the matrix of the input feature map and the convolution kernel. The two-dimensional matrix mapping operation is omitted through index calculation, so that no additional cache space is required to store temporary data, which can reduce the consumption of cache space by the rearrangement of the two-dimensional matrix during the calculation process, improve the calculation speed and minimize the storage space occupation.

[0090] In the above embodiment, in the process of performing implicit matrix multiplication calculations with the channel dimension as the accumulation direction, combined with the aforementioned optimization of the data storage structure of the input feature map and the convolution kernel, the calculation speed can be further accelerated. Since the vector data is stored in a manner that uses the channel dimension as the lowest dimension of data storage, the elements at the same spatial position in all channels in the memory are stored continuously and adjacently. Therefore, during the calculation process, all channel information at the required spatial position can be read in one memory access operation, thereby improving the hit rate of memory access, reducing the data reading waiting time, and further improving the calculation speed.

[0091] In an example of the above embodiment, the vectorized computing component of the Sunway SW26010Pro processor can also be used to accelerate the vector operations in implicit matrix multiplication.

[0092] In another embodiment, in order to improve the bandwidth of DMA memory access when the calculation results are written back, the data of the output plane matrix calculated by the slave core array is rearranged before being written back to the main memory. The rearrangement refers to exchanging and combining the data calculated by the slave core with other slave cores in the same dimension to increase the continuity of the data in the same dimension; when accessing memory, when the read data is located in adjacent positions in the cache, the memory access operation can be merged into a single, larger memory access operation to achieve merged memory access, reduce the number of memory accesses, and increase the memory access bandwidth.

[0093] In one example, for example Figure 2 As shown, the output vector 206 calculated from the kernel of a unit can be expressed as (1, bH o / 8,bW o , C o / 8), when writing back, it is necessary to access and write back according to the lowest dimension of storage. That is, when the lowest dimension of storage in this embodiment is the channel dimension, the data element written back for the first time is (1, bHo / 8,bW o , 1 / 8C o ), the data element written back for the second time is (1, bH o / 8,bW o , 2 / 8C o ), and so on. Since the length of the data written back in a single time is C o / 8, the amount of data written back is too small to fully utilize the DMA bandwidth, so the DMA write-back speed cannot be fully utilized.

[0094] Therefore, we can first convert the units from the kernel calculation results to (1, bH o / 8,bW o / 8, C o / 8)*8; then use RMA communication to share LDM data between slave cores, and exchange the slave core data in the same row in the slave core array, so that the data in the unit slave core is rearranged as (1, bH o / 8,bW o / 8, C o ); at this time, the data from the core is written back to the main memory through DMA, and the length of the data written back at a time is C o , which can make full use of DMA bandwidth and achieve high-speed write-back of calculation results.

[0095] The method provided in the embodiment of the present invention combines the hardware characteristics of a multi-core processor to optimize the calculation method of three-dimensional convolution in a targeted manner.

[0096] (1) By adopting a multi-level step-by-step calculation method, the original single-level three-dimensional convolution calculation with high complexity is converted into a multi-level calculation with simpler logic, which can reduce the complexity of the calculation steps at each level and make it more suitable for parallel processing, thereby significantly improving the efficiency of three-dimensional convolution calculation. In particular, in deep learning and scientific computing applications that require the calculation of large-scale three-dimensional convolutions, the calculation method provided by the present invention not only improves performance, but also improves the overall efficiency of the system by reducing energy consumption.

[0097] (2) By designing subtask splitting calculations, we can maximize the use of cache for calculations, improve memory access efficiency and program execution speed, and achieve high-speed calculation of three-dimensional convolution.

[0098] (3) By storing the convolution kernel data in the LDM and designing a matrix convolution iterative calculation mechanism, the convolution kernel data in the cache and a large amount of input feature map data can be reused, greatly reducing the reading of redundant data and improving the temporal and spatial locality of the data. The improvement of spatial locality means that when performing the convolution operation, the relevant data is already in the LDM; the improvement of temporal locality reduces the repeated loading and saving operations of data.

[0099] (4) By using multi-level buffering technology and asynchronous DMA memory access mode, the overlap of computing and memory access operations is achieved, minimizing the time spent waiting for data during the computing process.

[0100] (5) By rearranging the output data, the continuity of the data in the same dimension is increased, memory access is merged, and the DMA memory access bandwidth is improved.

[0101] (6) By splitting the large-scale matrix convolution calculation task and distributing it to multiple slave cores in the slave core array for parallel calculation, RMA communication is used to share LDM data between the slave cores to complete the rearrangement of the calculation result data. At the same time, vectorized computing components are used for acceleration to fully utilize hardware resources and improve the processor's overall computing performance for three-dimensional convolution.

[0102] In another embodiment of the present invention, a three-dimensional convolution parallel computing device based on a multi-core processor is also provided, and the device can be deployed on any device or platform with computing and processing capabilities. Figure 4 FIG. 1 is a schematic diagram showing a three-dimensional convolution parallel computing device based on a multi-core processor according to an embodiment. Figure 4 As shown, the device 400 is deployed on a platform with a multi-core processor, wherein a core group of the multi-core processor includes a master core and a slave core array with multiple slave cores, specifically including:

[0103] The acquisition module 401 is configured to acquire a three-dimensional convolution calculation task, comprising an input feature map; the input feature map has a plurality of plane matrices composed of a first dimension and a second dimension arranged in a direction of a third dimension;

[0104] The segmentation module 402 is configured to segment the input feature map into a plurality of sub-input feature maps in a direction parallel to the third dimension thereof according to the available memory space size of the slave core array of the multi-core processor, so as to obtain a three-dimensional convolution calculation subtask;

[0105] The three-dimensional matrix convolution preparation module 403 is configured to respond to the three-dimensional convolution calculation subtask, the main core obtains a convolution kernel, the convolution kernel is composed of a plurality of convolution kernel plane matrices, the main core takes a number of sub-input feature map plane matrices equal to the number of convolution kernel plane matrices contained in the convolution kernel from the sub-input feature map along the direction of the third dimension, and sends them together with the convolution kernel to the slave kernel array to perform three-dimensional matrix convolution calculation;

[0106] The three-dimensional matrix convolution calculation module 404 is configured to divide the three-dimensional matrix convolution calculation into a number of equal number of calculation units corresponding to the number of available slave cores in the slave core array, each of the slave cores is responsible for the calculation of one of the calculation units, and during the calculation process of the calculation unit, data exchange is performed in the memory of the slave core array, and the slave cores are calculated in parallel. After all the slave cores have completed the calculation, the results are written back to the main memory.

[0107] See also Figure 5 , an embodiment of the present invention provides a computing device 500, including a processor 510 and a memory 520. The memory 520 is used to store computer instructions. The processor 510 is used to execute the computer instructions stored in the memory 520, so that the computing device 500 can execute the above Figure 1 The method embodiment is shown.

[0108] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A three-dimensional convolution parallel computing method based on a multi-core processor, wherein a core group of the multi-core processor comprises a master core and a slave core array having a plurality of slave cores, characterized in that: The following steps are involved: Obtaining a three-dimensional convolution calculation task, including an input feature map; the input feature map has a plurality of planar matrices consisting of a first dimension and a second dimension arranged in a direction of a third dimension; According to the available memory space size of the slave core array of the multi-core processor, the master core divides the input feature map into a plurality of sub-input feature maps in a direction parallel to the third dimension thereof, to obtain a three-dimensional convolution calculation subtask; In response to the three-dimensional convolution calculation subtask, the main core obtains a convolution kernel, the convolution kernel is composed of a plurality of convolution kernel plane matrices, the main core takes a number of sub-input feature map plane matrices equal to the number of convolution kernel plane matrices contained in the convolution kernel from the sub-input feature map along the direction of the third dimension, and sends them together with the convolution kernel into the slave core array to perform three-dimensional matrix convolution calculation; The three-dimensional matrix convolution calculation is divided into a number of computing units of equal number corresponding to the number of available slave cores in the slave core array, each of the slave cores is responsible for the calculation of one of the computing units, and during the calculation process of the computing units, data exchange is performed in the memory of the slave core array, and the slave cores are calculated in parallel. After all the slave cores have completed the calculation, the results are written back to the main memory.

2. The method according to claim 1, characterized in that The input feature map and the convolution kernel both have channel dimension parameters, and the method further includes: The vector data of the input feature map and the convolution kernel are stored with the channel dimension as the lowest dimension.

3. The method according to claim 1, characterized in that The method further comprises: The data of the convolution kernel is preloaded into the memory of the slave kernel array and remains there until the three-dimensional convolution calculation task is completed.

4. The method according to claim 1, characterized in that: Writing the result back to the main memory comprises: The result data of the three-dimensional matrix convolution calculation obtained from the kernel array calculation is rearranged in the same dimension, and the rearranged data is written back to the main memory.

5. The method according to claim 4, characterized in that The method further comprises: Data is exchanged between the slave cores of the slave core array using RMA communication to complete the rearrangement of the result data of the three-dimensional matrix convolution calculation in the same dimension.

6. The method according to claim 1, characterized in that Each of the slave cores is responsible for the calculation of one of the computing units, including: The slave core calculates the computing unit using implicit matrix multiplication.

7. The method according to claim 1, characterized in that The method further comprises: Iteratively performing the three-dimensional matrix convolution calculation along the third dimension direction of the sub-input feature map until all plane matrices of the sub-input feature map are calculated; When the first iterative calculation is completed, within the data range required for the second iterative calculation, part of the loaded data continues to reside in the memory of the slave core array, and the remaining unread data is loaded into the memory of the slave core array.

8. The method according to claim 7, characterized in that The remaining unread data is loaded into the memory of the slave core array, including: The remaining unread data are loaded through asynchronous DMA mode.

9. A three-dimensional convolution parallel computing device based on a multi-core processor, deployed on a platform with a multi-core processor, wherein a core group of the multi-core processor includes a master core and a slave core array with multiple slave cores, characterized in that: include: An acquisition module: configured to acquire a three-dimensional convolution calculation task, comprising an input feature map; the input feature map has a plurality of plane matrices consisting of a first dimension and a second dimension arranged in a direction of a third dimension; A segmentation module: configured to divide the input feature map into a plurality of sub-input feature maps in a direction parallel to the third dimension thereof according to the available memory space size of the slave core array of the multi-core processor, so as to obtain a three-dimensional convolution calculation subtask; A three-dimensional matrix convolution preparation module: configured to respond to the three-dimensional convolution calculation subtask, the main core obtains a convolution kernel, the convolution kernel is composed of a plurality of convolution kernel plane matrices, the main core takes a number of sub-input feature map plane matrices equal to the number of convolution kernel plane matrices contained in the convolution kernel from the sub-input feature map along the direction of the third dimension, and sends them together with the convolution kernel into the slave core array to perform three-dimensional matrix convolution calculation; Three-dimensional matrix convolution calculation module: configured to divide the three-dimensional matrix convolution calculation into a number of equal number of calculation units corresponding to the number of available slave cores in the slave core array, each of the slave cores is responsible for the calculation of one of the calculation units, and during the calculation process of the calculation unit, data exchange is performed in the memory of the slave core array, and the slave cores are calculated in parallel. After all the slave cores have completed the calculation, the results are written back to the main memory.

10. A computing device, characterized in that The invention comprises a processor and a memory, wherein the processor is used to execute instructions stored in the memory to execute the method according to any one of claims 1 to 8.

Images