A GPU-based deep neural network inference method

Abstract

This invention relates to the field of artificial intelligence inference technology, specifically a GPU-based deep neural network inference method. The method includes: receiving input data and then allocating a continuous activation buffer in GPU memory that matches the computational order of the network layers; extracting data in batches and loading them into the starting position of the buffer; sequentially performing convolution operations, activation function transformations, and pooling downsampling operations on the GPU; writing the output feature maps of each layer sequentially to the corresponding positions in the activation buffer; and finally performing matrix multiplication in the fully connected layer and outputting the inference result. This invention optimizes GPU memory usage by storing intermediate results in continuous and matched activation buffers, reusing the same buffer space for data from each layer, reducing redundant memory addressing and read / write operations, and improving the smoothness of deep neural network inference computation on the GPU.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence reasoning technology, and in particular to a GPU-based deep neural network reasoning method. Background Technology

[0002] When traditional deep neural networks perform inference computations on GPUs, they allocate separate GPU memory space for intermediate computation results at different network layers. The loading location of input data batches is not fixed, and the output feature maps of operations such as convolution, pooling, and fully connected layers are all written to independent GPU memory regions. During network inference, the storage locations of intermediate results from each layer are distributed, and the GPU memory allocation method is dynamically adjusted according to the network layers.

[0003] Independently partitioning GPU memory regions leads to significant fragmentation of GPU memory. Reading and writing intermediate computation results requires switching between multiple discrete memory addresses, limiting memory bandwidth efficiency. The more network layers there are, the more complex the addressing operations of discrete memory regions become. Storing and retrieving multi-layer feature maps incurs additional memory operation overhead, and independent memory regions cannot reuse intermediate result storage space.

[0004] To address the problems with existing memory allocation and data storage methods, it is necessary to construct a continuous activation buffer in the GPU memory with a layout that matches the computation order of the network layers. Data is loaded in batches to the beginning of the activation buffer, and the computation results of each layer are written sequentially to the corresponding preset positions in the activation buffer, thereby changing the operation mode of storing intermediate results in a scattered manner. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a GPU-based deep neural network inference method.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a GPU-based deep neural network inference method, comprising: Receive input data to be processed, the input data containing the original information that needs to be used for reasoning and calculation; A contiguous activation buffer is allocated in the GPU's video memory to store intermediate calculation results of each layer of the deep neural network. The layout of the activation buffer matches the calculation order of the network layers. Extract one or more data batches from the input data, and load each data batch into the starting position of the activation buffer; On the GPU, based on the pre-compiled computing kernel, convolution operations are performed on the input data located at a specific position in the activation buffer to generate the output feature map of the convolutional layer; The output feature map of the convolutional layer is written into the region reserved for the convolutional layer in the activation buffer. The output feature map of the convolutional layer stored in the activation buffer is subjected to a nonlinear transformation by an activation function to produce a nonlinear activation result. The nonlinear activation result is used as the input to the pooling layer, and a downsampling operation is performed on the GPU to generate the output feature map of the pooling layer. Write the output feature map of the pooling layer into the activation buffer at a new location specified for subsequent layers; For the high-level feature map stored in the activation buffer after multiple aforementioned operations, matrix multiplication calculations of the fully connected layer are performed on the GPU. The calculation results of the fully connected layer are output as the final result of the deep neural network inference.

[0007] As a further aspect of the present invention, a contiguous activation buffer is allocated in the GPU's video memory, comprising: Analyze the topology of the target deep neural network to determine the input and output data sizes required for computation at each layer of the network; Calculate the ideal starting address and alignment requirements for the output data of each layer in the active buffer; Based on the needs of all layers, plan the total buffer size from the start address to the end address to ensure that data blocks of each layer are stored continuously and without overlap in the buffer; Request the GPU driver to allocate contiguous video memory space of the total buffer size and obtain a pointer to the contiguous video memory space.

[0008] As a further aspect of the present invention, extracting one or more data batches from the input data and loading each data batch into the starting position of the activation buffer includes: A preset batch processing size is determined, wherein the batch processing size defines the number of data samples processed in a single inference calculation; From a continuous input data stream, sequentially extract a number of sample data equal to the batch processing size; The extracted sample data is format-standardized to conform to the tensor shape and data type required by the convolution operation kernel; The standardized sample data is transferred directly from system memory to the beginning of the activation buffer via GPU direct memory access operations.

[0009] As a further aspect of the present invention, on a GPU, a convolution operation is performed on the input data located at a specific position in the activation buffer according to a pre-compiled computing kernel, including: The input data of the current batch is read from the starting position of the activation buffer, the input data being organized in the form of a four-dimensional tensor; Read the preloaded convolution kernel weight parameters from the GPU's constant memory; A GPU computing grid containing multiple thread blocks and threads is launched, and the input tensor and the convolution kernel weight parameters are assigned to different thread blocks for processing; Within each thread block, the threads cooperate to load the required input data blocks and weight blocks from video memory into shared memory; The thread performs multiply-accumulate operations on shared memory, completes the partial sum of the convolution operation, and writes the partial sum result back to the target address of the activation buffer.

[0010] As a further aspect of the present invention, the output feature map of the convolutional layer stored in the activation buffer is subjected to a nonlinear transformation by applying an activation function, including: Read the output feature map of the convolutional layer from the target address of the activation buffer; The activation function type used by the current layer of the deep neural network is determined. This activation function type is determined and fixed in the computing kernel during the network model loading stage or before the kernel starts, and there is no need to dynamically read the instruction at runtime. Start a GPU kernel function to allocate a GPU thread for each element in the output feature map of the convolutional layer; Each thread performs a nonlinear function calculation on the element value it is responsible for, according to the calculation instructions. The calculated element value is written back to the original target address of the activated buffer, overwriting the original linear output value.

[0011] As a further aspect of the present invention, the nonlinear activation result is used as input to the pooling layer, and a downsampling operation is performed on the GPU, including: Read the data that has undergone nonlinear transformation from the activation buffer; this data is the input feature map of the pooling layer. Determine the window size and sliding step size for the pooling operation; Create a GPU thread for each output position of the pooling operation; Each thread selects or calculates a representative value from the data within the window according to the max pooling or average pooling strategy within its corresponding input region. The calculated representative value is written into the activation buffer to a new position reserved for the pooling layer output, forming a feature map with reduced spatial dimensions.

[0012] As a further aspect of the present invention, matrix multiplication calculations of the fully connected layer are performed on the GPU on the high-level feature map stored in the activation buffer after multiple aforementioned operations, including: Read the output feature map of the last pooling layer or convolutional layer from the activation buffer; The read feature map is flattened in memory, converting it from a multidimensional tensor into a single-dimensional column vector; During the model loading phase, the weight matrix of the fully connected layer is transposed once and stored in the GPU memory. This makes the storage format of the weight matrix fully aligned with the memory access mode of the cublasSgemv function, enabling continuous merged memory access and improving the utilization of GPU memory bandwidth. This transposition is only performed once during loading and does not increase the runtime overhead during the inference phase. The highly optimized GPU matrix multiplication library function is invoked to calculate the product of the column vector and the weight matrix, thereby obtaining the output vector of the fully connected layer; Write the output vector of the fully connected layer back to the specified location in the activation buffer.

[0013] As a further aspect of the present invention, the step of launching a GPU computing grid containing multiple thread blocks and threads, and assigning the input tensor and the convolution kernel weight parameters to different thread blocks for processing, includes: The total number of threads required is determined based on the size of the output feature map of the convolutional layer. Based on the computing power of the GPU hardware and the configuration of the convolution operation kernel, the maximum number of threads that each thread block can accommodate is determined, and the total number of threads is divided into multiple thread blocks based on this. Based on the width, height, and channel dimensions of the input tensor, a three-dimensional mapping relationship is established between the thread block index and the spatial location of the output feature map; The computational task of sliding the convolutional kernel on the input tensor is allocated to each thread block according to the three-dimensional mapping relationship, so that each thread block is responsible for computing the result of a continuous region on the output feature map. Within each thread block, one or more specific elements on the output feature map that the thread is responsible for calculating are determined based on the thread's local index within the thread block. Through the mapping relationship and allocation method, all threads in the GPU computing grid can collaboratively complete the calculation of the entire output feature map.

[0014] As a further aspect of the present invention, the step of starting a GPU kernel function and allocating a GPU thread for each element in the output feature map of the convolutional layer includes: Calculate the total number of elements contained in the output feature map of the convolutional layer, which is obtained by multiplying the batch size, height, width and number of channels of the output feature map; A one-dimensional GPU thread grid is started based on the total number of elements, wherein the total number of threads in the one-dimensional GPU thread grid is greater than or equal to the total number of elements; Assign a globally unique linear index to each thread in the one-dimensional GPU thread grid; Establish a conversion relationship between the global linear index of the thread and the multidimensional coordinates of the elements in the output feature map of the convolutional layer. The multidimensional coordinates include the batch index, height direction index, width direction index, and channel index of the element. Based on the aforementioned conversion relationship, each thread determines the specific element in the output feature map of the convolutional layer that it needs to process, according to its global linear index. Each thread reads the original value of the element based on the memory address of the determined element in the activation buffer, and performs a nonlinear transformation calculation on the original value according to the calculation instructions corresponding to the activation function. Each thread writes the result of the nonlinear transformation calculation back to the memory address where the original element was located in the activation buffer.

[0015] As a further aspect of the present invention, the step of calling a highly optimized GPU matrix multiplication library function to calculate the product of the column vector and the weight matrix to obtain the output vector of the fully connected layer includes: The column vector obtained after the flattening operation is set as the left matrix in the matrix multiplication operation, and the dimension of the column vector is N times 1, where N is the number of features; The weight matrix of the fully connected layer that has been transposed is set as the right matrix in the matrix multiplication operation. The dimension of the weight matrix is ​​M multiplied by N, where M is the number of output neurons of the fully connected layer. Based on the dimension N of the left matrix multiplied by 1 and the dimension M of the right matrix multiplied by N, determine the dimension M of the result matrix that needs to be specified when calling the GPU matrix multiplication library function multiplied by 1. The pointer to the starting address of the left matrix data in video memory, the dimension information of the left matrix, the pointer to the starting address of the right matrix data in video memory, the dimension information of the right matrix, and the video memory address pointer used to store the result are passed as parameters to the highly optimized GPU matrix multiplication library function. The highly optimized GPU matrix multiplication library function organizes thread grids and memory accesses internally to perform the multiplication calculation of the left matrix and the right matrix, and directly writes the calculated result vector of M multiplied by 1 dimension into the specified position of the activation buffer. The result vector is the output vector of the fully connected layer.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are as follows: By allocating contiguous activation buffers in GPU memory that match the computational order of network layers, the storage areas for intermediate computation results from each layer of the neural network can form a unified, contiguous address space. This memory allocation corresponds to the network inference execution flow. When performing operations such as convolution, activation function transformation, and pooling downsampling, the GPU can complete the reading and writing of intermediate feature maps based on a fixed address sequence, avoiding multiple address location operations caused by discrete memory regions and reducing the overhead of ineffective operations during memory access. Contiguous memory space eliminates memory fragmentation; the storage and retrieval of intermediate computation results are completed within a unified address range, maintaining a stable data transfer path and reducing memory bandwidth loss caused by scattered storage. This makes the GPU's access to intermediate feature maps smoother and improves the space utilization of memory resources.

[0017] The extracted data is loaded in batches into the starting position of the activation buffer. Convolutional layer output feature maps, non-linear activation results, and pooling layer output feature maps are all written into the activation buffer at their respective preset positions according to the computation flow. During multi-layer network inference, all intermediate computation results are stored using the same activation buffer, eliminating the need to allocate separate GPU memory regions for different network layers. Input and output data for each layer's operations are transferred within the same continuous buffer. When performing multi-layer continuous operations, the GPU does not need to frequently switch read / write permissions between different GPU memory regions. The generation and storage of intermediate feature maps can be executed sequentially, simplifying the GPU memory allocation and release process. The storage of multi-layer feature maps relies on the same buffer space, effectively controlling GPU memory usage, improving GPU memory reuse efficiency, reducing GPU memory scheduling pressure during neural network inference, and enhancing the overall consistency of inference execution. Attached Figure Description

[0018] Figure 1 This is a flowchart of a GPU-based deep neural network inference method according to the present invention; Figure 2 This is a flowchart of the convolution operation; Figure 3 The curves showing the relationship between the number of GPU thread blocks and thread utilization and memory access time; Figure 4 A curve showing the relationship between GPU thread block configuration and execution efficiency; Figure 5 The curve shows the impact of GPU thread block size on matrix multiplication performance. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0021] See Figure 1 The system receives input data to be processed, which contains the original information for inference calculations. A contiguous activation buffer is allocated in the GPU's memory to store intermediate calculation results of each layer of the deep neural network. The layout of the activation buffer matches the calculation order of the network layers. One or more data batches are extracted from the input data, and each data batch is loaded into the starting position of the activation buffer. On the GPU, according to a pre-compiled computation kernel, convolution operations are performed on the input data located at a specific position in the activation buffer to generate the output feature map of the convolutional layer. The output feature map of the convolutional layer is written into the region reserved for the convolutional layer in the activation buffer. An activation function is applied to the output feature map of the convolutional layer stored in the activation buffer to perform a nonlinear transformation, generating a nonlinear activation result. This nonlinear activation result is used as the input to the pooling layer, and a downsampling operation is performed on the GPU to generate the output feature map of the pooling layer. The output feature map of the pooling layer is written into a new position specified for subsequent layers in the activation buffer. After multiple operations, matrix multiplication calculations are performed on the fully connected layer of the high-level feature map stored in the activation buffer on the GPU, and the calculation result of the fully connected layer is output as the final result of the deep neural network inference.

[0022] In one embodiment of the present invention, a deep neural network processes an image classification task. The input data is a color image with a height of 224 pixels, a width of 224 pixels, and 3 channels. The batch processing size is set to 32. The topology of the target deep neural network is analyzed to determine the input and output data sizes required for each layer of the network. For example, the output feature map size of the first convolutional layer is 112 pixels in height, 112 pixels in width, and 64 channels; the output feature map size of the pooling layer is 56 pixels in height, 56 pixels in width, and 64 channels; and the input vector dimension of the fully connected layer is 25088. The ideal starting address and alignment requirements of the output data of each layer in the activation buffer are calculated. The alignment requirements are set to 128 bytes based on the GPU memory access granularity. The ideal starting address is obtained by accumulating the output data size of the previous layer and aligning it upwards to the 128-byte boundary. Based on the requirements of all layers, the total buffer size from the starting address to the ending address is planned to ensure that the data blocks of each layer are stored continuously and without overlap in the buffer. The formula for calculating the total buffer size is: ; in: Indicates the total size of the activation buffer. Indicates the total number of network layers. Indicates the first The number of bytes occupied by the layer's output data is equal to the number of output elements of that layer multiplied by the number of bytes per element. For example, float32 is 4 bytes. Indicates the number of aligned bytes, for example ,symbol This indicates a rounding up operation, requesting contiguous memory space of the total buffer size from the GPU driver and obtaining a pointer to that contiguous memory space. The total buffer size B is calculated based on the output sizes of all operator layers in the network that need to store independent output data. Operator layers that need to store independent output data include, but are not limited to, convolutional layers and pooling layers. Operations such as activation functions that modify data in-place without creating new data storage layers do not occupy additional space in the activation buffer. When planning the total size of the activation buffer, only the sum of the aligned sizes of the output data of each operator such as convolutional layers and pooling layers needs to be accumulated. A preset batch size is determined, which defines the number of data samples processed in a single inference computation. From a continuous input data stream, a number of sample data equal to the batch size are sequentially extracted, for example, 32 frames are extracted from a video stream as a batch. The extracted sample data is then format-normalized to conform to the tensor shape and data type required by the convolutional kernel. The normalization process includes linearly scaling image pixel values ​​from an integer range of 0-255 to a floating-point range of 0.0-1.0, and organizing the data into a four-dimensional tensor with batch, height, width, and channel dimensions. Through GPU direct memory access operations, the format-normalized sample data is directly transferred from system memory to the beginning of the activation buffer. Direct memory access operations are executed asynchronously using the GPU's cudaMemcpyAsync function to avoid blocking the host thread.

[0023] In some embodiments, when analyzing the topology of a target deep neural network, for complex networks with branches or residual connections, the activation buffer layout needs to reserve independent space for the output of branch paths and plan additional areas at merging points to store the merging results. It is understood that a continuous layout of activation buffers avoids the overhead of inter-layer data copying. In specific implementations, compared to schemes using multiple scattered buffers, continuous activation buffers reduce memory fragmentation and improve cache utilization. Optionally, the batch size is dynamically adjusted according to the available memory capacity. For example, when processing high-resolution images, the batch size is reduced to 16 to ensure that the total size of the activation buffer does not exceed the memory limit. Format normalization processing includes data type conversion, such as converting uint8 integers to float32 floating-point numbers to adapt to the GPU's floating-point arithmetic units. In some embodiments, direct memory access operations improve transmission bandwidth by fixing host memory. Fixing host memory uses cudaMallocHost to allocate page-locked memory, which maximizes the speed of data copying from the host to the device. It can be understood that the starting position of the activation buffer is used to store batches of input data, and the output data of subsequent layers are stored sequentially. During inference, each layer reads input data from the buffer and writes output data back to the specified position in the buffer, thereby achieving efficient data flow in video memory.

[0024] In one embodiment of the present invention, image data with a batch size of 32 and an input size of 224x224x3 is processed, see [reference]. Figure 2 The system reads the input data of the current batch from the beginning of the activation buffer. The input data is organized in the form of a four-dimensional tensor, with the four dimensions being batch size, height, width, and channels, specifically [32, 224, 224, 3]. Preloaded convolutional kernel weight parameters are read from the GPU's constant memory. The convolutional kernel weight parameters have dimensions [3, 3, 3, 64], corresponding to the convolutional kernel height, width, number of input channels, and number of output channels. A GPU computing grid containing multiple thread blocks and threads is started, and the input tensor and convolutional kernel weight parameters are assigned to different thread blocks for processing. The total number of threads required is determined based on the size of the output feature map of the convolutional layer. The output feature map size of the convolutional layer is 32 batches, 112 height, 112 width, and 64 channels. The formula for calculating the total number of threads is: ; in: This represents the total number of threads required to perform the convolution operation. This indicates the batch size, which is 32 in this case. This indicates the height of the output feature map, which is 112 in this case. This indicates the width of the output feature map, which is 112 in this case. This represents the number of channels in the output feature map, which is 64 in this case. The parallel computation used in the convolution operation assigns an independent GPU thread to each element in the output feature map for computation. The total number of GPU threads required to execute this convolution operation is the total number of elements contained in the output feature map. The specific mapping relationship between GPU threads and output data can be designed according to hardware characteristics and performance optimization requirements. For example, one thread can be responsible for calculating multiple channel values ​​at the same spatial location, or multiple threads can collaborate to calculate a single output value. Different thread organization methods will affect the specific calculation of the total number of threads, but all are based on the core concept of completing convolution computation in parallel by dividing the data into thread grids and thread blocks. Based on the computing power of the GPU hardware and the configuration of the convolution operation kernel, the maximum number of threads that each thread block can accommodate is determined. For example, based on the CUDA architecture, the maximum number of threads per thread block is 1024. Based on this, the total number of threads is divided into multiple thread blocks. The system is divided into 25,088 thread blocks, each containing 1,024 threads. A three-dimensional mapping relationship is established between the thread block index and the spatial location of the output feature map based on the width, height, and channel dimensions of the input tensor. This mapping determines that each thread block is responsible for computing a three-dimensional region on the output feature map. The computation task of sliding the convolutional kernel across the input tensor is allocated to each thread block according to this three-dimensional mapping, ensuring that each thread block is responsible for computing the result of a continuous region on the output feature map. Within each thread block, the local index of the thread within the block determines one or more specific elements on the output feature map that the thread is responsible for computing; for example, one thread may be responsible for computing the value of an output pixel on one channel. Through this mapping relationship and allocation method, all threads in the GPU computing grid collaboratively complete the computation of the entire output feature map. Within each thread block, threads collaboratively load the required input data blocks and weight blocks from video memory into shared memory. The size of the input data block covers all the input pixel regions required to compute the output region of this thread block. The weight block contains all the convolution kernel parameters required for computation in this thread block. The thread performs multiply-accumulate operations on shared memory, completes the partial sum of the convolution operation, and writes the partial sum result back to the target address of the activation buffer. The target address is calculated from the batch, height, width, and channel coordinates of the output feature map.

[0025] In some embodiments, preloaded convolutional kernel weight parameters are read from the GPU's constant memory. These weight parameters are copied to constant memory before inference begins using CUDA's `cudaMemcpyToSymbol` function, leveraging the caching capabilities of constant memory to accelerate frequently accessed weights. Reading preloaded convolutional kernel weight parameters from the GPU's constant memory includes: before inference computation begins, determining whether the total amount of weight parameters for the current convolutional layer exceeds the GPU's constant memory capacity limit. If not, the weight parameters are preloaded into constant memory using a corresponding memory copy operation, such as CUDA's `cudaMemcpyToSymbol` function, to utilize its caching capabilities. If the total amount of weight parameters exceeds the constant memory capacity, the weight parameters are stored in the GPU's global memory. During convolution operations, thread blocks in the GPU computation grid read the required weight parameters from their global memory addresses, and may optionally load frequently accessed weight data blocks into the thread block's shared memory to optimize access efficiency. It is understandable that storing convolutional kernel weights in constant memory instead of global memory can reduce access latency and bandwidth pressure on global memory. Optionally, the thread blocks in the launched GPU computing grid can be organized in one-dimensional, two-dimensional, or three-dimensional form. For example, for the output feature map, a three-dimensional thread block grid organization can be used, with grid dimensions of (batch size / 2, output height / 7, output width / 7) and thread block dimensions of (2,7,7,2) to more intuitively map the output space. In some embodiments, when threads perform multiply-accumulate operations on shared memory, loop unrolling and data prefetching optimizations are used. Each thread reads multiple data elements from shared memory into registers and performs multiply-accumulate operations in a loop to reduce instruction overhead. It is understandable that using shared memory as a high-speed data exchange and temporary storage area within thread blocks can reuse input data and weight data that are accessed multiple times, significantly reducing the number of accesses to global memory. Optionally, the operation of writing parts and results back to the target address of the activation buffer can undergo a reduction operation before writing. For example, intermediate results can be quickly exchanged and accumulated between threads through the warpshuffle instruction within the thread block, and finally a single thread writes the complete convolution result back to global memory.

[0026] In one embodiment of the invention, the output feature map of the convolutional layer has a size of 32 batches, 112 heights, 112 widths, and 64 channels. The output feature map of the convolutional layer is read from the target address of the activation buffer. The target address is determined by the previous convolution calculation step and stored in a pointer variable. The activation function type used by the current layer of the deep neural network is determined. A GPU kernel function is started, and a GPU thread is allocated to each element in the output feature map of the convolutional layer. The total number of elements contained in the output feature map of the convolutional layer is calculated. Batch size of its output feature maps Output height Output width and number of output channels The product of. Where, This represents the total number of elements contained in the output feature map of the convolutional layer. The value here is 32. The value here is 112. The value here is 112. The value here is 64. Based on the total number of elements. Launch a one-dimensional GPU thread grid, where the total number of threads in the one-dimensional GPU thread grid is greater than or equal to the total number of elements. For example, if the total number of threads is 25,690,112, each thread in the one-dimensional GPU thread grid is assigned a globally unique linear index, ranging from 0 to 25,690,111. A conversion relationship is established between the global linear index of a thread and the multidimensional coordinates of elements in the output feature map of the convolutional layer. The multidimensional coordinates include the batch index, height index, width index, and channel index of the element. The conversion relationship is achieved through division and modulo operations. For example, channel index = global linear index % 64, width index = (global linear index / 64) % 112, height index = (global linear index / (64 * 112)) % 112, and batch index = global linear index / (64 * 112 * 112). Through the conversion relationship, each thread determines the specific element in the output feature map of the convolutional layer it needs to process based on its global linear index. Each thread reads the original value of the element based on the memory address of the determined element in the activation buffer, and performs nonlinear transformation calculation on the original value according to the calculation instructions corresponding to the activation function. For the modified linear unit function, the calculation operation is to determine whether the original value is greater than zero. If it is greater than zero, the original value is output; otherwise, zero is output. Each thread writes the result value obtained by the nonlinear transformation calculation back to the memory address where the original element is located in the activation buffer.

[0027] In some embodiments, the computation instructions corresponding to the activation function are retrieved from constant memory. These instructions can be function pointers or identifiers used to jump to the code segment of the corresponding activation function within the GPU kernel function. For example, identifier 1 represents the modified linear unit function, and identifier 2 represents the sigmoid function. Storing the computation instructions of the activation function in constant memory allows all threads to read and execute the same judgment logic at high speed, avoiding instruction cache invalidation. Optionally, the total number of threads in the started one-dimensional GPU thread grid can be rounded up to an integer multiple of the thread bundle size. In some embodiments, when establishing the conversion relationship between the global linear index and multidimensional coordinates, more efficient bitwise operations can be used instead of division and modulo operations. For example, when the height, width, and number of channels are powers of 2, the index can be obtained through shift and bitwise AND operations. Each thread independently processes the nonlinear transformation of one output element, with no data dependency between threads, thus achieving extremely high parallelism and fully utilizing the GPU's many-core computing resources. Optionally, the calculation instruction corresponding to the activation function can be a more complex function, such as a modified linear unit function with a leakage coefficient. Its calculation instruction includes a preset leakage coefficient α. The thread's operation is to determine whether the original value is greater than zero. If it is greater than zero, the original value is output; otherwise, the product of the original value and the leakage coefficient α is output.

[0028] See Figure 3In the performance analysis of GPU thread scheduling, thread utilization and memory access time exhibit a typical inverse correlation. Specifically, as the number of thread blocks gradually increases from 0 to 80,000, thread utilization (solid dot line) shows a pattern of first rapidly increasing and then stabilizing: the utilization is approximately 85% when the number of thread blocks is very low, continuously climbing between 10,000 and 60,000, reaching a peak of approximately 98.8% at 60,000, and then maintaining a high level above 98% between 60,000 and 80,000. This reflects the process of GPU computing resources being gradually and fully utilized as the size of the thread blocks increases. In contrast, memory access time (solid box) shows a monotonically decreasing and gradually converging characteristic as the number of thread blocks increases: in the initial stage (0 to 20000), memory access time drops rapidly from about 4.2μs to about 3.2μs, continues to slowly decrease to about 2.4μs in the range of 20000 to 60000, and stabilizes between 2.4μs and 2.5μs after 60000. This reflects that as the number of thread blocks increases, the GPU memory access mode is gradually optimized, and memory access conflicts and latency are effectively alleviated. From a performance synergy perspective, there is a strong coupling relationship between the improvement of thread utilization and the reduction of memory access time: when the number of thread blocks is insufficient, the idle rate of GPU computing units is high and memory access does not form an efficient batch mode, resulting in low utilization and long memory access latency; as the size of thread blocks expands to the optimal range (about 60,000), computing resources and memory bandwidth are fully matched, achieving both near-saturated thread utilization and compressing memory access time to the minimum level, achieving the optimal performance balance; beyond this range, further increasing the number of thread blocks has very little room for improvement in utilization, and memory access time also tends to stabilize due to scheduling overhead, reflecting the performance ceiling effect under the constraints of GPU hardware resources.

[0029] In one embodiment of the present invention, when processing an image classification task, after the pre-convolution and activation function calculations are completed, the data that has undergone nonlinear transformation is read from the activation buffer. This data is the input feature map of the pooling layer. The input feature map has a fixed four-dimensional shape, for example, a batch dimension of 32, a height dimension of 56, a width dimension of 56, and a channel dimension of 64. The window size and stride of the pooling operation are determined. Common pooling operations use a square window with a height of 2 pixels and a width of 2 pixels, and the stride is also 2 in both the height and width directions. A GPU thread is created for each output position of the pooling operation. The total number of output positions is determined by the input size, window size, and stride. The formula for calculating the total number of output positions is: ; in: This indicates the total number of output positions for the pooling operation, i.e., the total number of GPU threads that need to be created. Indicates batch size, Indicates the height of the input feature map. This indicates the width of the input feature map. This represents the number of channels in the input feature map. Indicates the height of the pooling window. Indicates the width of the pooling window. Indicates the sliding step size in the height direction. Indicates the sliding step size in the width direction, symbol This indicates a round-down operation. Based on the formula, the output feature map size is 32 batches, 28 height, 28 width, and 64 channels; therefore, it needs to be created... There are 12 GPU threads. Each thread, within its corresponding input region, selects or calculates a representative value from the data within the window using either max pooling or average pooling strategies. For max pooling, the thread compares four values ​​within its corresponding 2x2 input window and selects the maximum value. For average pooling, the thread calculates the arithmetic mean of the four values ​​within its corresponding 2x2 input window. The calculated representative value is written to a new location reserved for the pooling layer output in the activation buffer, forming a feature map with reduced spatial dimensions. The address offset of the new location in the activation buffer is pre-calculated and stored during buffer layout planning. The above calculation of 25088 thread blocks based on the pooling layer parameters represents an example of this specific network layer with a given batch size. In GPU parallel computing practice, the configuration of thread block size and number needs to balance computational resource utilization and scheduling overhead. In this example, the thread block size is set to 256, and the number of thread blocks is calculated based on the network output size. This configuration method aims to align with the generally efficient range of thread resource scheduling when GPU hardware performs convolution, pooling, and other operations. This method of calculation and configuration based on network parameters helps to achieve higher utilization of computing resources during inference. See Table 1.

[0030] Table 1: Feature map size and number of threads before and after pooling operation parameter Pooling layer input feature map Pooling layer output feature map illustrate Batch size 32 32 Pooling operations do not change the batch dimension high 56 28 Use a 2x2 window with a step size of 2 and a height halved. width 56 28 Use a 2x2 window with a step size of 2 and a width halved. Number of channels 64 64 Pooling operations are applied independently to each channel. Total number of GPU threads required - 1605632 Each output position is handled by a single thread. In some embodiments, the window size and stride of the pooling operation are determined. These parameters are read from the model configuration file as network hyperparameters; for example, the configuration file might specify `pooling_kernel=3` and `pooling_stride=2`. It is understood that the window size and stride determine the spatial dimensions of the output feature map and the downsampling rate, and are core parameters of the pooling operation. Optionally, a GPU thread is created for each output position of the pooling operation. Thread creation is achieved by calling a CUDA kernel function, with the kernel function configuring the thread block size to 256, and the grid size based on the total number of output positions. Calculated as In some embodiments, each thread selects or calculates a representative value within its corresponding input region according to a max-pooling or average-pooling strategy. The input region is obtained by inverse mapping of the output position coordinates, window size, and stride. For example, for the output position (… ), whose corresponding upper left corner coordinates are ( The thread reads the starting coordinates. Each input value is used for computation. It's understandable that since the computation at each output position is completely independent and the computational load is small, allocating a thread to each output position can achieve extremely high parallel granularity, fully utilizing the GPU's many-core architecture. Optionally, the operation of writing the calculated representative value to a new position in the activation buffer uses a merged write method, that is, consecutively numbered threads write to consecutive video memory addresses as much as possible to improve the utilization of video memory write bandwidth.

[0031] See Figure 4 In the analysis of the relationship between GPU thread block configuration and execution efficiency, the thread block size directly affects the utilization of GPU hardware resources and computational parallelism, thus determining the execution efficiency of inference tasks. The trend shows that execution efficiency exhibits a single-peak distribution, first increasing and then decreasing, with thread block size: when the thread block size is small (e.g., approximately 64), the execution efficiency is only 85.2%. At this point, the excessive number of thread blocks leads to increased thread scheduling overhead, and the resources of the GPU streaming multiprocessor (SM) are not fully utilized, resulting in insufficient parallelism. As the thread block size increases to around 256, the execution efficiency reaches a peak of 95.8%. Under this configuration, the number of thread blocks and the number of SMs are optimally matched, the utilization of hardware resources such as shared memory and registers reaches saturation, and the thread scheduling overhead is effectively distributed, achieving a balance between computational density and resource consumption. As the thread block size continues to increase (e.g., 512, 1024), the execution efficiency gradually drops to 92.3% and 88.1%, respectively. At this point, a single thread block consumes too many hardware resources, leading to a decrease in the number of concurrently active thread blocks, uneven load distribution between SMs, and intensified resource contention for shared memory and registers, thus reducing overall parallel efficiency. The curve reveals the core optimization principle of thread block configuration in GPU inference tasks: the thread block size must match the resource granularity of the GPU hardware architecture to achieve an optimal balance between resource utilization and scheduling overhead. In this experiment, 256 is the optimal thread block size for this scenario, maximizing the execution efficiency of deep neural network inference.

[0032] In one embodiment of the present invention, after multiple convolution, activation, and pooling operations, the output feature map of the last pooling layer has a size of 32 batches, 7 heights, 7 widths, and 64 channels. The output feature map of the last pooling layer is read from the activation buffer. The reading operation is completed by passing in the starting address pointer of the feature map stored in the activation buffer and the data size parameters. The read feature map is flattened in memory to convert it from a multidimensional tensor into a single-dimensional column vector. The flattening operation reshapes the tensor of shape [32,7,7,64] into a column vector of shape [10048,1], where 10048 is calculated from 32×7×7×64. During the model loading phase, the weight matrix of the fully connected layer is transposed once and stored in GPU memory. This ensures that the storage format of the weight matrix is ​​fully aligned with the memory access mode of the `cublasSgemv` function, enabling continuous merged memory access. This transposition is performed only once during loading and does not increase the runtime overhead during the inference phase. The original weight matrix with dimensions [10048, 1024] is transposed into a matrix with dimensions [1024, 10048]. A highly optimized GPU matrix multiplication library function is called to calculate the product of the column vector and the weight matrix, obtaining the output vector of the fully connected layer. The column vector obtained after the flattening operation is set as the left matrix in the matrix multiplication operation, with a dimension of N multiplied by 1, where N is the number of features (N=10048 in this case). The transposed weight matrix of the fully connected layer is set as the right matrix in the matrix multiplication operation, with a dimension of M multiplied by N, where M is the number of output neurons in the fully connected layer (M=1024 in this case). Based on the dimensions N multiplied by 1 for the left matrix and M multiplied by N for the right matrix, the result matrix must be multiplied by 1 (i.e., 1024 multiplied by 1) when calling the GPU matrix multiplication library function. The matrix multiplication formula is expressed as follows: ; in: This represents the output vector of the fully connected layer, with dimension . , This represents the weight matrix of the transposed fully connected layer, with dimension 1. , This represents the column vector obtained by flattening the feature map, with dimension . ,symbol This represents a matrix multiplication operation. A pointer to the starting address of the left matrix in GPU memory, the dimension information of the left matrix, a pointer to the starting address of the right matrix in GPU memory, the dimension information of the right matrix, and a pointer to the GPU memory address used to store the result are passed as parameters to a highly optimized GPU matrix multiplication library function. This library function can be the `cublasSgemv` function from the CUDA basic linear algebra subroutine library. The highly optimized GPU matrix multiplication library function internally organizes thread grids and memory accesses, performs the multiplication of the left and right matrices, and directly writes the calculated result vector (M multiplied by 1 dimension) directly to a specified location in the activation buffer. This result vector is the output vector of the fully connected layer, and the specified location is reserved when planning the activation buffer layout.

[0033] In some embodiments, the read feature map is flattened in memory. This flattening operation can be completed through a single data rearrangement copy within GPU memory, without changing the data's intrinsic values, only its logical shape descriptor. Flattening a multidimensional tensor into a one-dimensional column vector is understood to meet the shape requirements of fully connected layer matrix multiplication for input data, i.e., vector-matrix multiplication. Optionally, the weight matrix of the fully connected layer is read from GPU memory. The weight matrix can be stored in a contiguous, byte-aligned memory region to optimize memory merging efficiency for subsequent library function accesses. In some embodiments, a highly optimized GPU matrix multiplication library function is invoked. Internally, this library function automatically selects the optimal thread block size, grid size, and shared memory strategy based on the matrix dimension and GPU hardware architecture to perform matrix-vector multiplication. Using highly optimized standard library functions instead of implementing custom kernel functions ensures that the computation process fully utilizes the computing power and memory bandwidth of the specific GPU hardware, achieving a computation speed close to theoretical peak performance.

[0034] See Figure 5In the performance optimization of matrix multiplication in fully connected GPU layers, thread block size is a core configuration parameter determining computational efficiency and memory bandwidth utilization. This experiment systematically analyzes the impact of adjusting the thread block size on two key indicators: computation time and memory bandwidth. Computation time (solid dot): When the thread block size increases from its initial value to approximately 256, the computation time rapidly decreases from approximately 18.5ms to approximately 7.2ms, showing a significant performance improvement. When the thread block size continues to increase to 1024, the time only slowly rises to approximately 9.1ms, remaining at a low level overall. This trend indicates that moderately increasing the thread block size can improve GPU hardware parallelism and instruction issuance efficiency, but beyond the optimal threshold, resource contention and scheduling overhead will slightly offset the performance gains. Memory bandwidth (solid box): When the thread block size is 256, the memory bandwidth reaches a peak of approximately 820GB / s; as the thread block size further increases, the bandwidth decreases in a stepwise manner to approximately 750GB / s. This reflects that the optimal thread block size can maximize the merging of memory accesses and improve cache hit rate, thus approaching the theoretical memory bandwidth of the hardware. Excessively large thread blocks lead to excessively high shared memory usage and increased memory access conflicts, thereby reducing bandwidth utilization. Overall, a thread block size of approximately 256 is the optimal configuration point for this experiment. At this size, computation time is lowest and memory bandwidth is highest, achieving an efficient balance between computational resources and memory bandwidth, providing near-peak performance for matrix multiplication in the fully connected layer.

[0035] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A GPU-based deep neural network inference method, characterized in that, include: Receive input data to be processed, the input data containing the original information that needs to be used for reasoning and calculation; A contiguous activation buffer is allocated in the GPU's video memory to store intermediate calculation results of each layer of the deep neural network. The layout of the activation buffer matches the calculation order of the network layers. Extract one or more data batches from the input data, and load each data batch into the starting position of the activation buffer; On the GPU, based on the pre-compiled computing kernel, convolution operations are performed on the input data located at a specific position in the activation buffer to generate the output feature map of the convolutional layer; The output feature map of the convolutional layer is written into the region reserved for the convolutional layer in the activation buffer. The output feature map of the convolutional layer stored in the activation buffer is subjected to a nonlinear transformation by an activation function to produce a nonlinear activation result. The nonlinear activation result is used as the input to the pooling layer, and a downsampling operation is performed on the GPU to generate the output feature map of the pooling layer. Write the output feature map of the pooling layer into the activation buffer at a new location specified for subsequent layers; For the high-level feature map stored in the activation buffer after multiple aforementioned operations, matrix multiplication calculations of the fully connected layer are performed on the GPU. The calculation results of the fully connected layer are output as the final result of the deep neural network inference.

2. The GPU-based deep neural network inference method as described in claim 1, characterized in that, Allocate a contiguous activation buffer in the GPU's video memory, including: Analyze the topology of the target deep neural network to determine the input and output data sizes required for computation at each layer of the network; Calculate the ideal starting address and alignment requirements for the output data of each layer in the active buffer; Based on the needs of all layers, plan the total buffer size from the start address to the end address to ensure that data blocks of each layer are stored continuously and without overlap in the buffer; Request the GPU driver to allocate contiguous video memory space of the total buffer size and obtain a pointer to the contiguous video memory space.

3. The GPU-based deep neural network inference method as described in claim 2, characterized in that, Extracting one or more data batches from the input data and loading each data batch into the starting position of the activation buffer includes: A preset batch processing size is determined, wherein the batch processing size defines the number of data samples processed in a single inference calculation; From a continuous input data stream, sequentially extract a number of sample data equal to the batch processing size; The extracted sample data is format-standardized to conform to the tensor shape and data type required by the convolution operation kernel; The standardized sample data is transferred directly from system memory to the beginning of the activation buffer via GPU direct memory access operations.

4. The GPU-based deep neural network inference method as described in claim 3, characterized in that, On the GPU, convolution operations are performed on the input data located at a specific position in the activation buffer, based on a pre-compiled computation kernel, including: The input data of the current batch is read from the starting position of the activation buffer, the input data being organized in the form of a four-dimensional tensor; Read the preloaded convolution kernel weight parameters from the GPU's constant memory; A GPU computing grid containing multiple thread blocks and threads is launched, and the input tensor and the convolution kernel weight parameters are assigned to different thread blocks for processing; Within each thread block, the threads cooperate to load the required input data blocks and weight blocks from video memory into shared memory; The thread performs multiply-accumulate operations on shared memory, completes the partial sum of the convolution operation, and writes the partial sum result back to the target address of the activation buffer.

5. The GPU-based deep neural network inference method as described in claim 4, characterized in that, The output feature map of the convolutional layer stored in the activation buffer is subjected to a nonlinear transformation by applying an activation function, including: Read the output feature map of the convolutional layer from the target address of the activation buffer; The activation function type used by the current layer of the deep neural network is determined. This activation function type is determined and fixed in the computing kernel during the network model loading stage or before the kernel starts, and there is no need to dynamically read the instruction at runtime. Start a GPU kernel function to allocate a GPU thread for each element in the output feature map of the convolutional layer; Each thread performs a nonlinear function calculation on the element value it is responsible for, according to the calculation instructions. The calculated element value is written back to the original target address of the activated buffer, overwriting the original linear output value.

6. The GPU-based deep neural network inference method as described in claim 5, characterized in that, The nonlinear activation result is used as input to the pooling layer, and a downsampling operation is performed on the GPU, including: Read the data that has undergone nonlinear transformation from the activation buffer; this data is the input feature map of the pooling layer. Determine the window size and sliding step size for the pooling operation; Create a GPU thread for each output position of the pooling operation; Each thread selects or calculates a representative value from the data within the window according to the max pooling or average pooling strategy within its corresponding input region. The calculated representative value is written into the activation buffer to a new position reserved for the pooling layer output, forming a feature map with reduced spatial dimensions.

7. The GPU-based deep neural network inference method as described in claim 6, characterized in that, For the high-level feature maps stored in the activation buffer after multiple aforementioned operations, matrix multiplication calculations of the fully connected layer are performed on the GPU, including: Read the output feature map of the last pooling layer or convolutional layer from the activation buffer; The read feature map is flattened in memory, converting it from a multidimensional tensor into a single-dimensional column vector; During the model loading phase, the weight matrix of the fully connected layer is transposed once and stored in the GPU memory. This makes the storage format of the weight matrix fully aligned with the memory access mode of the cublasSgemv function, enabling continuous merged memory access and improving the utilization of GPU memory bandwidth. This transposition is only performed once during loading and does not increase the runtime overhead during the inference phase. The highly optimized GPU matrix multiplication library function is invoked to calculate the product of the column vector and the weight matrix, thereby obtaining the output vector of the fully connected layer; Write the output vector of the fully connected layer back to the specified location in the activation buffer.

8. The GPU-based deep neural network inference method as described in claim 7, characterized in that, The process of launching a GPU computing grid containing multiple thread blocks and threads, and assigning the input tensor and the convolution kernel weight parameters to different thread blocks for processing, includes: The total number of threads required is determined based on the size of the output feature map of the convolutional layer. Based on the computing power of the GPU hardware and the configuration of the convolution operation kernel, the maximum number of threads that each thread block can accommodate is determined, and the total number of threads is divided into multiple thread blocks based on this. Based on the width, height, and channel dimensions of the input tensor, a three-dimensional mapping relationship is established between the thread block index and the spatial location of the output feature map; The computational task of sliding the convolutional kernel on the input tensor is allocated to each thread block according to the three-dimensional mapping relationship, so that each thread block is responsible for computing the result of a continuous region on the output feature map. Within each thread block, one or more specific elements on the output feature map that the thread is responsible for calculating are determined based on the thread's local index within the thread block. Through the mapping relationship and allocation method, all threads in the GPU computing grid can collaboratively complete the calculation of the entire output feature map.

9. The GPU-based deep neural network inference method as described in claim 8, characterized in that, The step of starting a GPU kernel function to allocate a GPU thread for each element in the output feature map of the convolutional layer includes: Calculate the total number of elements contained in the output feature map of the convolutional layer, which is obtained by multiplying the batch size, height, width and number of channels of the output feature map; A one-dimensional GPU thread grid is started based on the total number of elements, wherein the total number of threads in the one-dimensional GPU thread grid is greater than or equal to the total number of elements; Assign a globally unique linear index to each thread in the one-dimensional GPU thread grid; Establish a conversion relationship between the global linear index of the thread and the multidimensional coordinates of the elements in the output feature map of the convolutional layer. The multidimensional coordinates include the batch index, height direction index, width direction index, and channel index of the element. Based on the aforementioned conversion relationship, each thread determines the specific element in the output feature map of the convolutional layer that it needs to process, according to its global linear index. Each thread reads the original value of the element based on the memory address of the determined element in the activation buffer, and performs a nonlinear transformation calculation on the original value according to the calculation instructions corresponding to the activation function. Each thread writes the result of the nonlinear transformation calculation back to the memory address where the original element was located in the activation buffer.

10. The GPU-based deep neural network inference method as described in claim 9, characterized in that, The step of calling highly optimized GPU matrix multiplication library functions to calculate the product of the column vector and the weight matrix to obtain the output vector of the fully connected layer includes: The column vector obtained after the flattening operation is set as the left matrix in the matrix multiplication operation, and the dimension of the column vector is N times 1, where N is the number of features; The weight matrix of the fully connected layer that has been transposed is set as the right matrix in the matrix multiplication operation. The dimension of the weight matrix is ​​M multiplied by N, where M is the number of output neurons of the fully connected layer. Based on the dimension N of the left matrix multiplied by 1 and the dimension M of the right matrix multiplied by N, determine the dimension M of the result matrix that needs to be specified when calling the GPU matrix multiplication library function multiplied by 1. The pointer to the starting address of the left matrix data in video memory, the dimension information of the left matrix, the pointer to the starting address of the right matrix data in video memory, the dimension information of the right matrix, and the pointer to the video memory address used to store the result are passed as parameters to the highly optimized GPU matrix multiplication library function. The highly optimized GPU matrix multiplication library function organizes thread grids and memory accesses internally to perform the multiplication calculation of the left matrix and the right matrix, and directly writes the calculated result vector of M multiplied by 1 dimension into the specified position of the activation buffer. The result vector is the output vector of the fully connected layer.

Images