Method for accelerating convolutional neural network by using matrix sparsity on multi-GPU platform

Abstract

The invention discloses a method for accelerating a convolutional neural network by using matrix sparsity on a multi-GPU platform, and the method comprises the following steps: determining a data scale F and a GPU number G, and starting an MGPUSimm kernel; the input data is loaded to a multi-level cache from a global memory of the MGPUSimm; non-zero value mask traversal is carried out on input data, and a mask graph set is obtained through GPU parallel; convolution calculation is carried out through the mask graph, and zero value data input is ignored, so that calculation is reduced; if a pooling layer exists behind the convolutional layer, subsequent pooling layer calculation is completed; and completing calculation of the remaining layers and outputting a result to a global memory. The invention aims to provide a method for accelerating a convolutional neural network by using matrix sparsity on a multi-GPU platform, and aims to solve the current situations that a sparse matrix based on a CPU or a single GPU platform is low in calculation efficiency in convolutional layer calculation of the convolutional neural network and the overall calculation time of the convolutional neural network is relatively long. The method comprises the following steps: adding a mask to remove zero value calculation so as to reduce the calculation amount, and then inversely deducing an input data position from a result mask to ignore the influence of a 0 value in a sparse matrix on the calculation of a convolutional layer; and meanwhile, multiple GPUs are used in the steps of forming a mask image set, calculating a convolutional layer, calculating a pooling layer and the like, so that the overall calculation efficiency of the convolutional neural network is improved in parallel.

Description

Technical Field

[0001] The present invention mainly relates to the field of multi-GPU architecture and deep learning acceleration technology, and in particular to a method for accelerating convolutional neural networks by utilizing matrix sparsity on a multi-GPU platform. Background Art

[0002] Convolutional Neural Networks (CNNs) are deep learning models that have achieved tremendous success in computer vision. Their design is inspired by the biological visual system and aims to mimic how humans process vision. Over the past few years, CNNs have achieved significant progress in image recognition, object detection, image generation, and many other fields, becoming a crucial component of computer vision and deep learning research.

[0003] A graphics processing unit (GPU) is a processor specifically designed for graphics and parallel computing. Because GPUs have thousands of cores and can process large amounts of data simultaneously while maintaining high throughput, they have become the hardware of choice for training deep learning models, rapidly processing large-scale matrix parallel operations. As the complexity of computing tasks and the amount of data increase, the computing power of a single GPU may not be able to meet demand. Therefore, multi-GPU architectures have emerged to provide greater computing power and storage capacity. However, in multi-GPU architectures, frequent data transfer and synchronization are required between different GPUs to collaboratively complete computing tasks. To prevent data copying and synchronization from affecting computing performance, efficient data transfer and synchronization strategies are required.

[0004] The convolution operation in a convolutional neural network refers to the process of sampling the convolution kernel on the feature map. The convolution process involves a large number of multiplication and addition operations, so reducing these operations is effective for acceleration. Network pruning and RELU activation are common operations in CNNs, which result in a large number of zero values ​​in the network. For feature maps, the proportion of zero values ​​evolves, and after many cycles in the deep layers of the network, this ratio can exceed 0.8. The pursuit of accuracy in CNNs leads to dozens of cycles, so a large amount of sparsity in the feature map is inevitable. However, calculating these zero values ​​is useless for the convolution result. In other words, if the calculation of zero values ​​can be skipped in the convolution operation, this will reduce the multiplication and addition operations. Some efforts have focused on reducing zero-value calculations in neural networks by designing new hardware architectures, and these architectures have achieved remarkable acceleration effects. However, compared with GPU-based solutions, new architectures based on Field Programmable Gate Arrays (FPGAs) or Application Specific Integrated Circuits (ASICs) have relatively long development cycles and higher hardware barriers. GPUs are widely used to accelerate CNNs due to their powerful parallel processing capabilities, such as large-scale matrix multiplication based on the highly optimized cuBLAS. Currently, sparse convolutions on GPUs are usually implemented based on sparse libraries such as cuSPARSE. For convolutional layers, feature maps and filters can be processed through cuSPARSE through steps such as expansion, compression, and sparse matrix calculation.

[0005] However, when implementing CNNs on GPUs, it is difficult to achieve satisfactory performance by exploiting the sparsity in convolution operations, and the speed improvement achieved is very limited compared to cuBLAS-based convolution methods. Therefore, Xu Weizhi et al. proposed a new sparse feature map storage format (Extended and Compressed Row, ECR) and designed an ECR-based convolution algorithm that skips the calculation of zero values ​​and reduces the amount of computation. The ECR method only requires a single access to global memory to complete expansion, compression, and sparse matrix calculations. They also proposed a storage format that combines convolution and pooling operations (Pooling-pack Extended and Compressed Row, PECR), which not only exploits sparsity but also calculates the convolution layer and pooling layer together, reducing data transmission between the CPU and GPU and the transmission from the GPU's off-chip memory to the on-chip memory. Compared to cuDNN and cuSPARSE, their optimization method achieves speedups of 1.97x, 2.23x, 2.74x, and 1.58x on mainstream CNN models such as VGG-19, ResNet-50, DenseNet-121, and RegNetX-16GF, respectively. However, because Xu Weizhi et al.'s optimization method incorporates a new storage format, while significantly accelerating sparse matrix processing, it requires tedious data preprocessing to convert the input data to the new storage format, which still consumes a significant amount of time. Summary of the Invention

[0006] The purpose of this invention is to provide a method for accelerating convolutional neural networks by utilizing matrix sparsity on a multi-GPU platform, aiming to solve the current situation where sparse matrices are not efficient in convolutional neural network convolution layer calculations based on CPU or single GPU platforms, and the overall calculation time of convolutional neural networks is relatively long.

[0007] The present invention is implemented as follows: a method for accelerating convolutional neural networks using matrix sparsity on a multi-GPU platform comprises the following steps: Step 1: Determine the data size F and the number of GPUs G, and start the MGPUSim kernel; Step 2: Load the input data from MGPUSim's global memory into the multi-level cache; Step 3: Perform mask traversal of non-zero values ​​on the input data and obtain a set of mask maps in parallel through the GPU; Step 4: Perform convolution calculations through the mask image, ignoring zero-value data inputs, thereby reducing calculations; Step 5: If there is a pooling layer after the convolutional layer, complete the subsequent pooling layer calculation; Step 6: Complete the calculation of the remaining layers and output the results to the global memory.

[0008] Furthermore, the step 1 obtains the input data scale F (in MB) to be calculated and the number of GPUs used G (in units), and then initializes the MGPUSim environment and starts the MGPUSim kernel according to the number G.

[0009] Furthermore, the step 2 requires loading the input data from the global memory of MGPUSim into a multi-level cache, where the input data can be an image or a feature map.

[0010] Furthermore, in step 3, the input data is masked and traversed for non-zero values, and a mask map set is obtained in parallel by the GPU:

[0011] Step 1: Use multiple GPUs in parallel to calculate the mask of the non-zero value of the input data, that is, the relative coordinate δ of the non-zero value in the sliding window, by combining the sliding window size and step size. A non-zero value may have multiple relative coordinates, and these relative coordinates are stored in a two-dimensional array to form a mask map set;

[0012] Step 2: Use the designed multy_kernel function to calculate the product of all non-zero values ​​in the sparse matrix and the non-zero value of the convolution kernel in parallel on multiple GPUs, and store the product values ​​obtained by multiplying all non-zero values ​​with the non-zero value in the convolution kernel into a two-dimensional array for subsequent index accumulation calculation.

[0013] Furthermore, the step 4 performs convolution calculation through the mask image, ignoring zero-value data input, thereby reducing calculation.

[0014] Furthermore, the step five completes the subsequent pooling layer calculation.

[0015] Furthermore, step six completes the calculation of the remaining layers and outputs the results to the global memory.

[0016] The beneficial effects of the present invention are as follows: (1) The present invention is highly parallelized as a whole, and uses multi-GPU parallelism in the steps of forming a mask map set, convolution layer calculation, and pooling layer calculation, thereby improving the computational efficiency of the convolutional neural network in the case of sparse matrices. (2) The present invention can improve the overall execution efficiency without adding data preprocessing steps or modifying the original format of the data. (3) Although the present invention increases the space complexity by adding a mask, the storage advantage of a multi-GPU structure such as MGPUSim can offset the negative impact of the increased space complexity, thereby achieving the goal of exchanging space for time efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of a method for accelerating a convolutional neural network using matrix sparsity on a multi-GPU platform provided by an embodiment of the present invention;

[0018] Figure 2 This is a flowchart for accelerating the parallelization of convolutional neural networks by utilizing matrix sparsity based on MGPUSim, provided by an embodiment of the present invention;

[0019] Figure 3 is the output vector calculation formula. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0021] Figure 1 The process of the method for accelerating convolutional neural networks by utilizing matrix sparsity on a multi-GPU platform of the present invention is shown. As shown in the figure, the present invention is implemented as follows: a method for accelerating convolutional neural networks by utilizing matrix sparsity on a multi-GPU platform includes the following steps: S101: Determine the data scale F and the number of GPUs G, and start the MGPUSim kernel; S102: Load input data from MGPUSim's global memory into a multi-level cache; S103: Perform mask traversal of non-zero values ​​on the input data and obtain a mask map set in parallel through the GPU; S104: performing convolution calculations through the mask image, ignoring zero-value data inputs, thereby reducing calculations; S105: If there is a pooling layer after the convolution layer, then complete the subsequent pooling layer calculation; S106: Complete the calculation of the remaining layers and output the results to the global memory.

[0022] Furthermore, the S101 obtains the input data size F (in MB) to be calculated and the number of GPUs used G (in units), and then initializes the MGPUSim environment and starts the MGPUSim kernel according to the number G.

[0023] Furthermore, the S102 needs to load input data from the global memory of MGPUSim into a multi-level cache, where the input data may be an image or a feature map.

[0024] Furthermore, in step three, the input data is subjected to mask traversal of non-zero values, and a mask map set is obtained in parallel by the GPU.

[0025] Step 1: Use multiple GPUs in parallel to calculate the mask of the non-zero value of the input data, that is, the relative coordinate δ of the non-zero value in the sliding window, by combining the sliding window size and step size. A non-zero value may have multiple relative coordinates, and these relative coordinates are stored in a two-dimensional array to form a mask map set;

[0026] Step 2: Use the designed multy_kernel function to calculate the product of all non-zero values ​​in the sparse matrix and the non-zero value of the convolution kernel in parallel on multiple GPUs, and store the product values ​​obtained by multiplying all non-zero values ​​with the non-zero value in the convolution kernel into a two-dimensional array for subsequent index accumulation calculation.

[0027] Furthermore, the S104 performs convolution calculations through the mask image, ignoring zero-value data input, thereby reducing calculations.

[0028] Furthermore, the S105 completes the subsequent pooling layer calculation.

[0029] Furthermore, the S106 completes the calculation of the remaining layers and outputs the results to the global memory.

[0030] Example: MGPUSim is a highly configurable multi-GPU simulator, open source under the terms of the MIT License. It was developed using the Golang programming language, chosen for its reasonable performance and ease of programming and maintenance. Users can configure various simulator parameters, including the number of GPU cores, memory hierarchy, and scheduling policy. These parameters can be dynamically changed during simulation, allowing users to evaluate different scenarios and configurations for optimal performance and energy efficiency. The convolution operation in a convolutional neural network refers to the process by which the convolution kernel samples the feature map. The convolution process involves a large number of multiplications and additions, so reducing these operations is effective for acceleration. Network pruning and Relu activation are common operations in CNNs, which result in a large number of zero values ​​in the network. The proportion of zero values ​​in feature maps can evolve, exceeding 0.8 after many cycles in deep layers of the network. The pursuit of accuracy in CNNs can lead to dozens of cycles, making significant sparsity in feature maps unavoidable. However, calculating these zero values ​​is useless for the convolution results. In other words, if the calculation of zero values ​​can be skipped in the convolution operation, this will reduce the multiplication and addition operations. Therefore, using matrix sparsity to accelerate convolutional neural networks on multi-GPU platforms such as MGPUSim will have better performance. The steps to implement the method of using matrix sparsity to accelerate convolutional neural networks based on MGPUSim are as follows:

[0031] Step 1: Determine the data size F and the number of GPUs G, and start the MGPUSim kernel; We used the CIFAR-10 / 100 lightweight dataset, pre-processing the images to 224×224. We also used the classic VGG-16 convolutional model, with the convolutional layers using Submanifold Sparse CNN sparse convolutional layers. The number of GPUs, G, is specified as a parameter at runtime; here, we use four GPUs. After completing the above settings, we launched the MGPUSim kernel.

[0032] Step 2: Load the input data from MGPUSim's global memory into the multi-level cache; Use the code shown above to load input data from MGPUSim's global memory into a multi-level cache. This allows you to reuse the cached data in calculations and reduce global memory access.

[0033] Step 3: Perform mask traversal of non-zero values ​​on the input data and obtain a set of mask maps in parallel through the GPU; The stride is set to 1 and the convolution kernel size is 3×3. The mask of the non-zero values ​​of the input data, i.e., the relative coordinate δ of the non-zero value within the sliding window, is calculated in parallel using multiple GPUs, combining the sliding window size and stride. A non-zero value may have multiple relative coordinates, which are stored in a two-dimensional array to form a mask map set. The designed multy_kernel function is then used to calculate the product of all non-zero values ​​in the sparse matrix with the non-zero values ​​of the convolution kernel in parallel using multiple GPUs. The resulting products of all non-zero values ​​and the non-zero values ​​in the convolution kernel are stored in a two-dimensional array for subsequent index accumulation calculations.

[0034] Step 4: Perform convolution calculations through the mask image, ignoring zero-value data inputs to reduce calculations; use the designed out_kernel function to calculate each time the sliding window and the convolution kernel are multiplied in parallel on multiple GPUs, such as Figure 3 The difference between the original coordinate X of the non-zero element in the window shown in the original input data and the relative coordinate δ in the mask, or sliding window, corresponds to the output coordinate Y of the value in the output matrix. For elements with the same output coordinates, the relative coordinates of the element determine whether there is a corresponding multiplication value and then find the column of the two-dimensional array where the value is located. Then, based on the horizontal and vertical coordinates of the element, the row of the two-dimensional array where the value of the product of the element and the convolution kernel is located is found, that is, a unique value in the two-dimensional array is found. Finally, the values ​​of the product of all the elements with the same output coordinates and the convolution kernel are accumulated and output.

[0035] Step 5: If there is a pooling layer after the convolutional layer, complete the subsequent pooling layer calculation; We place a max pooling layer after each convolutional layer. This layer receives the feature map from the convolutional layer and slides the pooling window starting from the top left corner of the input feature map. The pooling window stride is set to 1, and the window size is set to 3×3 to ensure that the sliding window covers the entire input area. The output of the pooling sliding window is the maximum value in the window, calculated in parallel using multiple GPUs. After completion, the output feature map is passed to the next layer.

[0036] Step 6: Complete the calculation of the remaining layers and output the results to global memory; The fully connected layer receives the feature map passed by the pooling layer for calculation. The ReLU activation function is used in the hidden layer. After all calculations are completed, the results are output to the global memory.

[0037] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without the need for creative work are still within the scope of protection of the present invention.

Claims

1. A method for accelerating convolutional neural networks using matrix sparsity on a multi-GPU platform, characterized in that: The method for accelerating convolutional neural networks using matrix sparsity on a GPU platform includes the following steps: Step 1: Determine the data size F and the number of GPUs G, and start the MGPUSim kernel; Step 2: Load the input data from MGPUSim's global memory into the multi-level cache; Step 3: Perform mask traversal of non-zero values ​​on the input data and obtain a set of mask maps in parallel through the GPU; Step 4: Perform convolution calculations through the mask image, ignoring zero-value data inputs, thereby reducing calculations; Step 5: If there is a pooling layer after the convolutional layer, complete the subsequent pooling layer calculation; Step 6: Complete the calculation of the remaining layers and output the results to the global memory.

2. The method for accelerating convolutional neural networks using matrix sparsity on a multi-GPU platform according to claim 1, wherein: The step 1 obtains the input data size F (in MB) to be calculated and the number of GPUs used G (in units), and then initializes the MGPUSim environment and starts the MGPUSim kernel according to the number G.

3. The method for accelerating convolutional neural networks using matrix sparsity on a multi-GPU platform according to claim 1, wherein: The step 2 requires loading the input data from the global memory of MGPUSim into a multi-level cache, where the input data can be an image or a feature map.

4. The method for accelerating convolutional neural networks using matrix sparsity on a multi-GPU platform according to claim 1, wherein: Step 3 requires inputting data to perform mask traversal of non-zero values, and obtaining a mask map set in parallel through the GPU. The specific steps are as follows: Step 1: Use multiple GPUs in parallel to calculate the mask of the non-zero value of the input data, that is, the relative coordinate δ of the non-zero value in the sliding window, by combining the sliding window size and step size. A non-zero value may have multiple relative coordinates, and these relative coordinates are stored in a two-dimensional array to form a mask map set; Step 2: Use the designed multy_kernel function to calculate the product of all non-zero values ​​in the sparse matrix and the non-zero value of the convolution kernel in parallel on multiple GPUs, and store the product values ​​obtained by multiplying all non-zero values ​​with the non-zero value in the convolution kernel into a two-dimensional array for subsequent index accumulation calculation.

5. The method for accelerating convolutional neural networks using matrix sparsity on a multi-GPU platform according to claim 1, wherein: Step 4 requires using the designed out_kernel function to calculate, in parallel across multiple GPUs, the difference between the original coordinate X in the original input data and the relative coordinate δ in the mask (i.e., the sliding window) of each non-zero element in the window, each time the sliding window is multiplied by the convolution kernel. The corresponding coordinate difference is the output coordinate Y of that value in the output matrix. For elements with the same output coordinates, the relative coordinates of the element are used to determine whether there is a corresponding multiplication value, and then the column of the two-dimensional array containing that value is found. Then, based on the element's horizontal and vertical coordinates, the row of the two-dimensional array containing the product of the element and the convolution kernel is found, thus finding a unique value in the two-dimensional array. Finally, the product values ​​of all elements with the same output coordinates and the convolution kernel are accumulated and output.

6. The method for accelerating convolutional neural networks using matrix sparsity on a multi-GPU platform according to claim 1, wherein: Step 5 mentioned above requires the completion of subsequent pooling layer calculations, and the specific steps are as follows: Step 1: If there is a pooling layer after the convolutional layer, it will receive the feature map from the convolutional layer. The pooling window starts from the upper left corner of the input feature map and slides in sequence according to the step size. The step size of the pooling layer does not necessarily need to be consistent with the convolutional layer depending on the situation, ensuring that the sliding window can cover the entire input area. Step 2: Use multiple GPUs to perform a maximum pooling operation on the area covered by the window. After the calculation is completed, the output feature map is passed to the next layer. If there is no subsequent layer and it is the last pooling layer, the output feature map is output to global memory.

7. The method for accelerating convolutional neural networks using matrix sparsity on a multi-GPU platform according to claim 1, wherein: The step 6 needs to receive the feature map formed in step 5, perform activation function calculation and linear layer calculation, and output the results to the global memory until all layers are calculated.

Images