A method for inference acceleration of a GPU-based sparse deep neural network

Abstract

The application provides a GPU-based inference acceleration method of a sparse deep neural network, which is used for accelerating an inference process of the sparse deep neural network on to-be-processed data, the GPU comprises a global memory, the sparse deep neural network comprises multiple network layers, network layer weight parameters of the sparse deep neural network transmitted from a CPU are stored in the global memory, and the method comprises the following steps: S1, based on a size of the global memory, a required memory size of the to-be-processed data as a whole and a required size of single-layer network layer weight parameters, block processing is performed on the to-be-processed data according to a preset block rule to obtain block data, so that each block data can be completely stored in the global memory; and S2, each block data is calculated with the network layer weight parameters stored in the global memory until each block data and each network layer weight parameter of the sparse deep neural network complete calculation.

Description

Technical Field

[0001] This invention relates to the field of deep learning, specifically to inference acceleration technology for sparse deep neural networks in the field of deep learning, and more specifically to GPU-based inference acceleration technology for sparse deep neural networks, i.e., a GPU-based inference acceleration method for sparse deep neural networks. Background Technology

[0002] Deep Neural Networks (DNNs) have been widely applied in various fields such as natural language processing, autonomous driving, and image enhancement due to their powerful data mining and pattern analysis capabilities, achieving excellent results. To improve the accuracy of deep neural network models, researchers have continuously increased the number of parameters, resulting in a number of large-scale deep neural network models with over 100 million parameters. For example, the Generative Pre-Trained Transformer-3 (GPT-3) model contains 175 billion parameters. Another example is the Bidirectional Encoder Representation from Transformers (BERT) model, which contains 340 million parameters. While increasing the number of parameters in a deep neural network model can improve its accuracy, the extremely high demand for computing and storage resources resulting from this significant increase in parameters leads to a substantial increase in equipment costs and energy consumption.

[0003] To reduce costs and save energy, researchers have proposed pruning the weight parameters in large-scale deep neural network models to decrease the number of weight parameters and computational load, while minimizing the impact on performance. Although pruning can reduce the number of weight parameters in large-scale deep neural network models to some extent, the resulting sparse deep neural network model still has a very large number of parameters. This means that a large number of weight parameters will still be involved in the calculation when performing inference tasks. Taking a sparse deep neural network model with L layers as an example, the calculation process of the weights in a single layer is briefly explained. If the activation function of this model uses the most commonly used Rectified Linear Unit (ReLU) function, then the weights of the L-th layer in the sparse deep neural network model... The computation process of a layer network can be represented as follows: ,in, Indicates the first The network layer outputs data. Indicates the first The input data and output data of the layer network are both dense matrices; Indicates the first The layer weights of the network are sparse matrices. This is the bias matrix, usually a constant matrix. As can be seen from the calculation process, the core computation of the sparse deep neural network model is the sparse-dense matrix multiplication (SpMM) process.

[0004] Because the number of parameters in the sparse deep neural network model obtained after pruning is still enormous, a graphics processing unit (GPU) is typically used to accelerate the sparse deep neural network model when performing inference tasks. This acceleration is achieved by improving the computation process of sparse-dense matrix multiplication. The reason for using a GPU to improve the inference speed of the sparse deep neural network model is its powerful parallel processing and floating-point arithmetic capabilities. The streaming multiprocessor (SM) in the GPU is responsible for scheduling and executing computational tasks in units of thread bundles. The computational resources included are scalar computation cores and tensor cores. Scalar computation cores can perform scalar computations of different precisions in parallel, while tensor cores support direct computation of matrix multiplications of specific sizes.

[0005] To improve the inference speed of sparse deep neural network models, existing technologies typically set up inference acceleration schemes for GPU-based sparse deep neural networks from the following aspects: (1) Storage format optimization, i.e., designing a new sparse matrix storage format, compressing non-zero elements by row or column, and ensuring load balance among the threads of the graphics processing unit (GPU) by data partitioning; (2) Parallel mode selection, designing parallel mode optimization space for the triple loop in sparse-dense matrix multiplication (SpMM), and selecting the optimal parallel scheme by combining performance models and actual running data tests; (3) Utilizing new computing resources, on the basis of making the sparse matrix locally dense by performing elementary row and column transformations, using tensor cores to calculate the matrix multiplication of the dense part; (4) Input data and network layer weight parameters partitioning, transmitting the input data in blocks and batches to the graphics processing unit (GPU), and transmitting the network layer weight parameters of the sparse deep neural network to the graphics processing unit (GPU) asynchronously.

[0006] While existing technologies can improve the inference speed of sparse deep neural network models, they are typically designed for high-end data center graphics processing units (GPUs) and cannot be adapted to resource-constrained low-end GPUs. As a result, when sparse deep neural networks run on resource-constrained low-end GPUs, issues such as memory overflow, low running speed, and low throughput may occur. Summary of the Invention

[0007] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a GPU-based inference acceleration method for sparse deep neural networks and a GPU-based inference acceleration device for sparse deep neural networks.

[0008] The objective of this invention is achieved through the following technical solutions.

[0009] According to a first aspect of the present invention, a GPU-based method for accelerating inference of a sparse deep neural network is provided, for accelerating the inference process of the sparse deep neural network on data to be processed. The GPU includes global memory, and the sparse deep neural network includes multiple network layers. The global memory stores the network layer weight parameters of the sparse deep neural network transmitted from the CPU. The method includes: S1, dividing the data to be processed into blocks according to a preset block division rule based on the size of the global memory, the total memory required for the data to be processed, and the required size of the weight parameters of a single network layer, so that each block of data can be completely stored in the global memory; S2, calculating each block of data with the network layer weight parameters stored in the global memory, until the calculation of each block of data with each network layer weight parameter of the sparse deep neural network is completed.

[0010] In some embodiments of the present invention, the global memory is provided with a first buffer and a second buffer, which respectively store the network layer weight parameters corresponding to a preset number of layers. The first buffer and the second buffer are both configured to store the network layer weight parameters in the following manner: after all the network layer weight parameters stored in the current buffer are calculated with the currently participating block data, all the network layer weight parameters stored in the other buffer continue to be calculated with the intermediate results of the calculation of the currently participating block data and all the network layer weight parameters stored in the current buffer. The current buffer then re-stores other network layer weight parameters that were not involved in the calculation and were transmitted by the CPU.

[0011] In some embodiments of the present invention, the preset number of layers is 1.

[0012] In some embodiments of the present invention, the preset number of layers corresponding to each buffer is determined in the following manner:

[0013]

[0014]

[0015]

[0016] in, This indicates the number of network layers stored in each buffer. This indicates the time it takes for the CPU to transfer the weight parameters of a single network layer to the GPU. This indicates that the number of row vectors is The data and the calculation time of single-layer network weight parameters, This represents the number of row vectors in the block data.

[0017] In some embodiments of the present invention, when the sum of the preset number of layers corresponding to the first buffer and the second buffer is less than the number of network layers of the sparse deep neural network, the number of network layers re-stored in each buffer is calculated as follows:

[0018]

[0019] in, This indicates the number of the last network layer stored in the current buffer. This indicates the number of the first network layer stored in another buffer. Indicates the number of inputs in the current buffer. Non-zero row vectors of data in a layer network. Indicates the input in another buffer. Non-zero row vectors of data in a layer network. The non-zero row vector is represented as Data and the first The computation time of layer weight parameters in a multilayer network. This indicates the time it takes for the CPU to transfer the weight parameters of a single network layer to the GPU. This indicates the number of network layers currently being restored in the buffer.

[0020] In some embodiments of the present invention, in step S1, when the sum of the total memory size required for the data to be processed and the memory size of the weight parameters of the two single-layer network layers stored in the block compression matrix column format is greater than or equal to the global memory size of the GPU, the data to be processed is divided into blocks according to the preset block division rules.

[0021] In some embodiments of the present invention, the preset segmentation rule is as follows:

[0022]

[0023]

[0024]

[0025] in, Indicates the size of the data blocks to be processed. This indicates the number of network layers stored in each buffer. This indicates the memory size of the auxiliary array used for processing the data. This indicates the global memory size of the GPU.

[0026] In some embodiments of the present invention, the GPU includes a scalar computation core and a tensor core, which are configured to: when it is necessary to perform matrix operations on each block of data and each network layer weight parameter, divide each network layer weight parameter along the column vector to the scalar computation core and the tensor core according to a preset division ratio, and have the scalar computation core and the tensor core perform matrix operations on the current network layer weight parameter and the block of data in parallel according to their respective proportions.

[0027] According to a second aspect of the present invention, an inference acceleration device for a GPU-based sparse deep neural network is provided. The sparse deep neural network includes multiple network layers. The device includes an input device, a communication bus, a memory, a CPU, a GPU, and an output device, wherein: the input device is used to receive input control information and transmit it to the CPU; the communication bus is used for connection and data communication between the input device, the memory, the CPU, the GPU, and the output device; the memory is used to store the network layer weight parameters and data to be processed of the sparse deep neural network; the CPU is used to receive and parse the control information and then transmit the data to be processed and the network layer weight parameters of the sparse deep neural network to the GPU through the communication bus; the GPU is used to perform inference on the data to be processed according to the method described in the first aspect of the present invention to obtain an inference result; and the output device is used to display the inference result after the GPU performs inference on the data to be processed.

[0028] Compared with the prior art, the advantages of the present invention are: (1) Based on the device environment of the GPU and the information of the sparse deep neural network model, the block size of the input data is modeled as an optimization problem to solve the partition size of the input data and the number of layers to allocate the weight parameters of the network layer. At the same time, the number of layers to allocate the weight parameters of the network layer is adjusted in real time during the inference execution stage to make full use of the storage resources in the resource-constrained GPU; (2) Utilizing the concurrency capabilities of the scalar computing core and the tensor core, the weight parameters of the network layer are allocated to the scalar computing core and the tensor core according to the preset partition ratio. The scalar computing core and the tensor core perform matrix operations on the corresponding proportion of the weight parameters of the network layer and the input data in parallel, so that all cores can participate in the inference acceleration and thus improve the computational throughput of the inference process of the sparse deep neural network model. Attached Figure Description

[0029] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0030] Figure 1 This is a flowchart illustrating a GPU-based inference acceleration method for sparse deep neural networks according to an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the composition of a GPU-based sparse deep neural network inference acceleration device according to an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0033] As mentioned in the background section, although existing solutions can improve the running speed of sparse deep neural network models inference, these solutions are usually designed for high-end data center graphics processing units (GPUs) and cannot be adapted to low-end GPUs with limited resources. Therefore, when sparse deep neural networks run on low-end GPUs with limited resources, they may experience storage overflow, low running speed, and low throughput.

[0034] To address the aforementioned issues, the inventors analyzed existing inference acceleration schemes for sparse deep neural network models based on GPU architectures and discovered deficiencies in both storage and computational resource scheduling. Specifically, regarding storage resource scheduling, existing technologies typically determine the input data block size and the number of layers for allocating network layer weight parameters based solely on empirical values. This increases the debugging cost of migrating sparse deep neural network models to new graphics processing unit (GPU) architectures. Regarding computational resource scheduling, existing technologies generally utilize only one type of computational resource—either the scalar or tensor cores of the GPU—to accelerate the sparse-dense matrix multiplication (SpMM) computation process, leaving the other computational resource idle. This failure to fully utilize the computational resources in the GPU hinders the improvement of execution efficiency in the sparse deep neural network inference process. Based on this, the inventors proposed a two-pronged approach to accelerate inference for sparse deep neural networks based on GPU architecture. Firstly, leveraging the GPU's device environment and the sparse deep neural network model information, the input data block size is modeled as an optimization problem to determine the partition size of the input data and the number of layers for allocating network layer weight parameters. Simultaneously, during the inference execution phase, an online algorithm is used to adjust the number of layers for allocating network layer weight parameters in real time, fully utilizing the storage resources of the resource-constrained GPU. Secondly, utilizing the concurrency capabilities of scalar and tensor cores, network layer weight parameters are allocated to scalar and tensor cores according to a preset partitioning ratio. The scalar and tensor cores then perform parallel matrix operations on their respective proportions of network layer weight parameters and input data, ensuring that all cores can participate in inference acceleration and thus improving the computational throughput of the sparse deep neural network model's inference process.

[0035] To better understand the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0036] In summary, such as Figure 1 As shown, this invention provides a GPU-based method for accelerating inference of sparse deep neural networks, used to speed up the inference process of the sparse deep neural network on data to be processed. The GPU includes global memory, and the sparse deep neural network includes multiple network layers. The global memory stores the network layer weight parameters of the sparse deep neural network transmitted from the CPU. The method includes: S1, based on the size of the global memory, the total memory required for the data to be processed, and the required size of the weight parameters of a single network layer, dividing the data to be processed into blocks according to a preset block division rule to obtain block data, so that each block data can be completely stored in the global memory; S2, calculating each block data with the network layer weight parameters stored in the global memory, until the calculation of each block data with each network layer weight parameter of the sparse deep neural network is completed.

[0037] To better understand the proposed solution, the following section provides a detailed explanation of the sparse deep neural network inference acceleration process of this invention, covering aspects such as network layer weight parameter storage, data block processing, and computational resource allocation.

[0038] I. Storage of Network Layer Weight Parameters

[0039] The network layer weight parameters of the sparse deep neural network are stored in the global memory of the GPU. According to one embodiment of the present invention, the global memory includes a first buffer and a second buffer, which respectively store the network layer weight parameters corresponding to a preset number of layers. Both the first and second buffers are configured to store the network layer weight parameters as follows: after all network layer weight parameters stored in the current buffer have been calculated with the currently participating data block, all network layer weight parameters stored in the other buffer are further calculated with the intermediate results obtained from the calculation of the currently participating data block and all network layer weight parameters stored in the current buffer. The current buffer then re-stores other network layer weight parameters that were not involved in the calculation and were transferred by the CPU. It should be noted that, in order to reduce the storage resource consumption of the global memory, the network layer weight parameters are compressed before being stored in the first or second buffer. According to one example of the present invention, the weight parameters of a single-layer network are compressed as follows: Non-zero values ​​in each row and column of the weight matrix corresponding to the network layer weight parameters are compared; rows and columns with similar non-zero values ​​are moved closer together to generate a locally dense matrix; and the locally dense matrix is ​​compressed according to the Blocked Compressed Sparse Column (BCSR) format. It should also be noted that other compressed storage formats can be used to store the network layer weight parameters, and the present invention does not impose any special limitations.

[0040] According to one embodiment of the present invention, the preset number of layers is 1.

[0041] According to one embodiment of the present invention, the number of preset layers corresponding to each buffer is determined in the following manner:

[0042]

[0043]

[0044]

[0045] in, This indicates the number of network layers stored in each buffer. This indicates the time it takes for the CPU to transfer the weight parameters of a single network layer to the GPU. This indicates that the number of row vectors is The data and the calculation time of single-layer network weight parameters, This represents the number of row vectors in the data chunks. It should be noted that... and All data were obtained during the training phase of sparse deep neural network models.

[0046] According to one embodiment of the present invention, when the sum of the preset number of layers corresponding to the first buffer and the second buffer is less than the number of layers in the sparse deep neural network, the number of network layers re-stored in each buffer is calculated as follows:

[0047]

[0048]

[0049] in, This indicates the number of the last network layer stored in the current buffer. This indicates the number of the first network layer stored in another buffer. Indicates the number of inputs in the current buffer. Non-zero row vectors of data in a layer network. Indicates the input in another buffer. Non-zero row vectors of data in a layer network. The non-zero row vector is represented as Data and the first The computation time of layer weight parameters in a multilayer network. This indicates the time it takes for the CPU to transfer the weight parameters of a single network layer to the GPU. This indicates the number of network layers currently being restored in the buffer.

[0050] II. Data Block Processing

[0051] The purpose of data chunking is to avoid GPU memory overflow, so that the inference process of sparse deep neural networks can proceed smoothly even when the total memory required for the data to be processed exceeds the resource-constrained global memory of the GPU.

[0052] According to one embodiment of the present invention, when the sum of the total memory required for the data to be processed and the memory size of the weight parameters of the two single-layer network layers stored in a block compression matrix column format is greater than or equal to the global memory size of the GPU, the data to be processed is divided into blocks according to a preset block division rule. It should be noted that the total memory required for the data to be processed and the memory size of the weight parameters of the two single-layer network layers stored in a block compression matrix column format are estimated maximum memory values ​​based on actual conditions. Using the estimated maximum memory value for block division can avoid memory overflow caused by insufficient memory reservation and block division errors. It should also be noted that when determining whether the data to be processed needs to be divided into blocks, the preset number of layers corresponding to the first buffer and the second buffer is 1. The purpose of this setting is to reserve as much storage space as possible for the data to be processed to reduce the need for block division. However, when GPU storage resources are relatively sufficient, the preset number of layers can be adjusted according to actual conditions and is not limited to 1.

[0053] According to one embodiment of the present invention, the preset segmentation rule is as follows:

[0054]

[0055]

[0056]

[0057] in, Indicates the size of the data blocks to be processed. This indicates the number of network layers stored in each buffer. This indicates the memory size of the auxiliary array used for processing the data. This represents the global memory size of the GPU. It should be noted that when the data to be processed needs to be divided into blocks, in order to obtain the largest possible block size (…), To reduce the number of data transfers in chunks, the number of network layers stored in the first and second buffers in the initial state is set to the minimum value of 1, and the network layer weight parameters that were not involved in the calculation are re-stored in the subsequent inference process.

[0058] According to an example of the present invention, the number of network layers initially stored in the first buffer and the second buffer is 1, and during the inference process, both the first buffer and the second buffer are configured to store network layer weight parameters in the following manner: after the single-layer network layer weight parameter (the i-th layer) stored in the current buffer is calculated with the currently participating block data, the single-layer network layer weight parameter (the i+1-th layer) stored in the other buffer continues to be calculated with the intermediate result after the calculation of the currently participating block data and all network layer weight parameters stored in the current buffer, and the current buffer re-stores the next layer (the i+2-th layer) network layer weight parameter of the network layer stored in the other buffer transmitted by the CPU.

[0059] According to an example of the present invention, the first and second buffers initially store 1 network layer, and during inference, both the first and second buffers are configured to store network layer weight parameters as follows: after all network layer weight parameters stored in the current buffer are calculated with the currently participating block data, the single-layer network layer weight parameters stored in the other buffer continue to be calculated with the intermediate results of the calculation of the currently participating block data and all network layer weight parameters stored in the current buffer. The current buffer then... The number of network layers that can be stored in the current buffer is recalculated to restore the network layer weight parameters that were not included in the calculation and were transmitted by the CPU. It should be noted that in practice, during block-based judgment and block-based processing, the initial number of network layers stored in the first and second buffers can be set to other data, not limited to 1. When the initial number of network layers stored in the first and second buffers can be set to other data, the storage principle of the first and second buffers during inference is the same as in the previous example, and will not be repeated here.

[0060] It should be noted that when the sum of the total memory required for the data to be processed and the memory required to store the weight parameters of the two single-layer network layers in a block-compressed matrix format is less than the GPU's global memory size, there is no need to divide the data into blocks. The absence of block division indicates that the total memory required for the data to be processed may be small, or the GPU's global memory may be large. In this case, after storing the entire data to be processed in global memory, there may still be considerable storage space remaining to store the network layer weight parameters. In this situation, [the data can be divided into blocks]. (at this time This represents the number of row vectors in the data to be processed that do not require partitioning. It also calculates the number of network layers that can be stored in each buffer in its initial state. If, in the initial state, the GPU's global memory can only fully store the data to be processed, and cannot store all the network layer weight parameters in the sparse deep neural network through the first and second buffers, then the number of network layers stored in the first and second buffers in their initial state is determined by... The calculated network layer weight parameters, which were not involved in the calculation, are then re-stored during subsequent inference processes and transmitted by the CPU.

[0061] According to one example of the invention, by The number of network layers stored in the first and second buffers in their initial states is calculated. During inference, both the first and second buffers are configured to store network layer weight parameters as follows: after all network layer weight parameters stored in the current buffer are calculated with the currently participating block data, all network layer weight parameters stored in the other buffer continue to be calculated with the intermediate results of the calculation of the currently participating block data and all network layer weight parameters stored in the current buffer. The current buffer then restores the next layer weight parameters of the last network layer stored in the other buffer, which are transmitted by the CPU.

[0062] According to one example of the invention, by The number of network layers stored in the first and second buffers in their initial states is calculated. During inference, both the first and second buffers are configured to store network layer weight parameters as follows: After all network layer weight parameters stored in the current buffer are calculated with the currently used data block, all network layer weight parameters stored in the other buffer are further calculated with the intermediate results of the calculations using the currently used data block and all network layer weight parameters stored in the current buffer. The current buffer is then processed through... The number of network layers that can be stored in the current buffer is recalculated in order to restore the network layer weight parameters that were not involved in the calculation and were transferred by the CPU. It should be noted that by adjusting the number of network layers that can be stored in the buffer in real time during inference, it is possible to ensure that while intermediate results are being calculated with the network layer weight parameters in one buffer, the CPU is transferring new network layer weight parameters to another buffer. This achieves overlap between calculation and data transfer, thereby avoiding a decrease in GPU processing performance.

[0063] III. Allocation of Computing Resources

[0064] As mentioned in the background technology, existing technologies generally only utilize one type of computing resource—either the scalar computing core or the tensor core of the graphics processing unit (GPU)—to accelerate the computation process of sparse-dense matrix multiplication (SpMM), leaving the other computing resource idle. This fails to fully utilize the computing resources in the graphics processing unit and is not conducive to improving the execution efficiency of the sparse deep neural network inference process.

[0065] To fully utilize the computing resources in the GPU and improve computational throughput, according to one embodiment of the present invention, the scalar computing core and tensor core are configured such that, when matrix operations involving each block of data and each network layer weight parameter need to be performed, each network layer weight parameter is allocated along the column vector to the scalar computing core and tensor core according to a preset partitioning ratio. The scalar computing core and tensor core then perform parallel matrix operations on their respective proportions of the current network layer weight parameters and block of data. It should be noted that the preset partitioning ratio is the ratio of the maximum concurrent computational workload of the scalar computing core and tensor core obtained during the training of the sparse deep neural network. It should also be noted that the scalar computing core and tensor core perform parallel matrix operations on their respective proportions of the current network layer weight parameters and block of data using multiple thread bundles. Specifically, for the data allocated to the scalar computation cores (the corresponding proportion of the current network layer weight parameters and block data), the data needs to be further partitioned and distributed to all scalar computation core thread bundles. The amount of data in each scalar computation core thread bundle must ensure that each thread bundle can handle the update of at least a single input matrix element. If this cannot be achieved, all computational tasks need to be transferred to the tensor cores. Similarly, for the data allocated to the tensor cores (the corresponding proportion of the current network layer weight parameters and block data), the data needs to be further partitioned and distributed to all tensor core thread bundles. The amount of data in each tensor core thread bundle must ensure that each thread bundle can handle computations of a specific data size, for example, where both the column and row dimensions are multiples of 16.

[0066] Inference acceleration methods based on the aforementioned sparse deep neural networks, such as Figure 2As shown, the present invention also provides an inference acceleration device for a GPU-based sparse deep neural network, wherein the sparse deep neural network includes multiple network layers. The device includes an input device, a communication bus, a memory, a CPU, a GPU, and an output device, wherein: the input device is used to receive input control information and transmit it to the CPU; the communication bus is used for connection and data communication between the input device, the memory, the CPU, the GPU, and the output device; the memory is used to store the network layer weight parameters and data to be processed of the sparse deep neural network; the CPU is used to receive control information, parse it, and then transmit the data to be processed and the network layer weight parameters of the sparse deep neural network to the GPU through the communication bus; the GPU is used to perform inference on the data to be processed according to the method described in the foregoing embodiments to obtain an inference result; the output device is used to display the inference result after the GPU performs inference on the data to be processed.

[0067] The beneficial effects of the present invention are as follows: (1) Based on the device environment of the GPU and the information of the sparse deep neural network model, the block size of the input data is modeled as an optimization problem to solve the partition size of the input data and the number of layers to allocate the weight parameters of the network layer. At the same time, the number of layers to allocate the weight parameters of the network layer is adjusted in real time during the inference execution stage to make full use of the storage resources in the resource-constrained GPU; (2) Utilizing the concurrency capabilities of the scalar computing core and the tensor core, the weight parameters of the network layer are allocated to the scalar computing core and the tensor core according to the preset partition ratio. The scalar computing core and the tensor core perform matrix operations on the corresponding proportion of the weight parameters of the network layer and the input data in parallel, so that all cores can participate in the inference acceleration and thus improve the computational throughput of the inference process of the sparse deep neural network model.

[0068] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0069] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0070] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0071] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A GPU-based inference acceleration method for sparse deep neural networks, used to accelerate the inference process of the sparse deep neural network on data to be processed, wherein the GPU includes global memory, and the sparse deep neural network includes multiple network layers, characterized in that, The method includes storing network layer weight parameters of the sparse deep neural network transmitted from the CPU in the global memory. S1. Based on the global memory size, the total memory size required for the data to be processed, and the required size for the weight parameters of a single network layer, the data to be processed is divided into blocks according to a preset block division rule to obtain block data, so that each block data can be completely stored in the global memory. S2. Calculate each data block with the network layer weight parameters stored in the global memory until the calculation of each data block with the weight parameters of each network layer of the sparse deep neural network is completed. The global memory includes a first buffer and a second buffer, which store the network layer weight parameters corresponding to a preset number of layers, respectively. Both the first and second buffers are configured to store the network layer weight parameters in the following manner: After all the network layer weight parameters stored in the current buffer have been calculated with the currently participating block data, all the network layer weight parameters stored in the other buffer continue to be calculated with the intermediate results of the calculation of the currently participating block data and all the network layer weight parameters stored in the current buffer. The current buffer then re-stores the other network layer weight parameters that were not involved in the calculation and were transmitted by the CPU. The preset number of layers corresponding to each buffer is determined in the following way: in, This indicates the number of network layers stored in each buffer. This indicates the time it takes for the CPU to transfer the weight parameters of a single network layer to the GPU. This indicates that the number of row vectors is The data and the calculation time of single-layer network weight parameters, This represents the number of row vectors in the partitioned data; Wherein, when the sum of the preset number of layers corresponding to the first buffer and the second buffer is less than the number of layers in the sparse deep neural network, the number of network layers re-stored in each buffer is calculated as follows: in, This indicates the number of the last network layer stored in the current buffer. This indicates the number of the first network layer stored in another buffer. Indicates the number of inputs in the current buffer. Non-zero row vectors of data in a layer network. Indicates the input in another buffer. Non-zero row vectors of data in a layer network. The non-zero row vector is represented as Data and the first The computation time of layer weight parameters in a multilayer network. This indicates the time it takes for the CPU to transfer the weight parameters of a single network layer to the GPU. This indicates the number of network layers currently being restored in the buffer.

2. The method according to claim 1, characterized in that, The preset number of layers is 1.

3. The method according to claim 1, characterized in that, In step S1, when the sum of the total memory size required for the data to be processed and the memory size of the weight parameters of the two single-layer network layers stored in the block compression matrix column format is greater than or equal to the global memory size of the GPU, the data to be processed is divided into blocks according to the preset block division rules.

4. The method according to claim 3, characterized in that, The preset segmentation rule is as follows: in, Indicates the size of the data blocks to be processed. This indicates the number of network layers stored in each buffer. This indicates the memory size of the auxiliary array used for processing the data. This indicates the global memory size of the GPU.

5. The method according to claim 1, characterized in that, The GPU includes scalar computation cores and tensor cores, which are configured as follows: When it is necessary to perform matrix operations on each block of data and each network layer weight parameter, each network layer weight parameter is divided along the column vector to the scalar calculation core and the tensor core according to a preset division ratio. The scalar calculation core and the tensor core then perform matrix operations on the current network layer weight parameter and the block of data in parallel according to their respective proportions.

6. An inference acceleration device for a GPU-based sparse deep neural network, wherein the sparse deep neural network comprises multiple network layers, characterized in that, The device includes an input device, a communication bus, a memory, a CPU, a GPU, and an output device, wherein: The input device is used to receive input control information and transmit it to the CPU; The communication bus is used for connection and data communication between input devices, memory, CPU, GPU and output devices; The memory is used to store the network layer weight parameters and data to be processed of the sparse deep neural network. The CPU is used to receive control information, parse it, and then transmit the data to be processed and the network layer weight parameters of the sparse deep neural network to the GPU through the communication bus. The GPU is used to perform inference on the data to be processed in accordance with the method described in any one of claims 1-5 to obtain an inference result; The output device is used to display the inference results after the GPU performs inference on the data to be processed.

7. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1 to 5.

Images