A neural network-enhanced graphics processor pipeline architecture

Abstract

The present invention relates to a neural network-enhanced graphics processor pipeline architecture. The neural network-enhanced graphics processor pipeline architecture of the present invention optimizes the GPU pipeline architecture for neural network computing acceleration scenarios, performs neural network-related enhancements on the traditional pipeline, expands a high-speed accessible global weight register array dedicated to weights, and designs a specially optimized register mapping method to avoid performance losses and power consumption overhead caused by repeated transfer of weights between computing components and shared memory, or even global memory. This achieves significant improvement in neural network computing performance at a small cost in area while maintaining graphics processing capabilities.

Description

Technical Field

[0001] The present invention relates to the field of graphics processor technology, and in particular to a neural network enhanced graphics processor pipeline architecture. Background Art

[0002] A graphics processing unit (GPU) is a computer component specifically designed to handle graphics-related computing tasks. Compared to a CPU, a GPU allocates more computing resources to the computational portion rather than the control portion, and maximizes the use of parallelism in graphics computing tasks. As a result, it can perform graphics processing tasks much faster than a CPU. A GPU generally includes a large number of identical or similar processing elements (PEs), which are grouped at different levels for easier management. A common approach is to group a certain number of PEs that use the same shared memory or texture unit, and call the resulting module a stream multiprocessor (SM) or a compute unit (CU).

[0003] A neural processing unit (NPU) typically refers to a computing component that accelerates neural network computations. Convolutional neural networks are currently the mainstream type of neural network, and neural networks have been a major area of AI development in recent years. Furthermore, neural networks are primarily deep neural networks with multi-layered structures. Consequently, there are many similar, but slightly different, terms like artificial intelligence accelerator (AI accelerator), convolutional neural network accelerator (CNNA), and deep learning accelerator (DLA).

[0004] Although it is common to use GPUs for neural network calculations on desktops and servers, this usually involves using additional tensor computing cores or adding low-precision compact computing units to the GPU's computing components. When performing neural network calculations, operations such as convolution are converted into general matrix-matrix multiplications, leveraging the high parallelism of GPU operations for acceleration.

[0005] On the other hand, on the mobile side, due to limitations such as area and power consumption, the GPU is usually dedicated to graphics-related processing, and an additional NPU is embedded in the system on a chip (SoC) to perform neural network-related calculations, and some operators that are difficult for the NPU to support are handed over to the CPU for calculation.

[0006] However, simply increasing computational parallelism without leveraging data locality in neural network computations often makes memory access a bottleneck, while additional tensor computing cores and NPUs undoubtedly require a larger chip area, which means increased cost and design complexity. Summary of the Invention

[0007] To this end, the technical problem to be solved by the present invention is to provide a neural network enhanced graphics processor pipeline architecture for neural network computing acceleration scenarios and improving neural network computing performance.

[0008] To solve the above technical problems, the present invention provides a neural network enhanced graphics processor pipeline architecture, which includes:

[0009] A stream multiprocessor, the stream multiprocessor comprising at least one processing unit, an instruction fetch and decode unit, and a memory access control unit; all processing units in the stream multiprocessor share the instruction fetch and decode unit; all processing units in the stream multiprocessor share the memory access control unit and access a data bus under the control of the memory access control unit. The neural network enhanced graphics processor pipeline architecture further comprises:

[0010] A weight read and write control unit, a global weight register array, and a weight mapping control unit, wherein the weight mapping control unit is provided in each processing unit, the weight read and write control unit is connected to the data bus, and all processing units in the stream multiprocessor share the weight read and write control unit and the global weight register array;

[0011] The weight read and write control unit is used to read the weight data required for the current convolution calculation into the global weight register array before the calculation; the weight mapping control unit is used to map the target part of the weight data stored in the global weight register array to the local mapping of the weight register of the current processing unit according to the current calculation status during the calculation process; the weight read and write control unit is also used to read the weight data required for the next convolution calculation into the cache of the global weight register array in advance.

[0012] In one embodiment of the present invention, the global weight register array is a multi-bank memory, and each bank can store a weight value of a filter.

[0013] In one embodiment of the present invention, a specific weight value may be determined by its bank address, row address, and column address.

[0014] In one embodiment of the present invention, a global controller is further included. All processing units in the stream multiprocessor share the global controller and are uniformly controlled by the global controller.

[0015] In one embodiment of the present invention, a shared storage unit is further included, and all processing units in the streaming multiprocessor share the shared storage unit under the control of the memory access control unit.

[0016] In one embodiment of the present invention, the weight mapping control unit is configured to map the target portion of the global weight register to the register space of the own processing unit according to the current calculation state during the calculation process, including:

[0017] The weight mapping control unit is used to fix partial values of the bank address, row address, and column address according to the convolution parameters calculated by the current processing unit during the calculation process, and decode another partial value of the address from the instruction, thereby mapping the target part of the global weight register array to the register space of the current processing unit.

[0018] The present invention also provides a method for performing neural network calculations on a graphics processor, which is applied to the above-mentioned neural network-enhanced graphics processor pipeline architecture, and includes the following steps:

[0019] S101: Initialize and configure the parallel mode and hyperparameter information of the weight reading and writing control unit and the weight mapping control unit;

[0020] S102, the weight read and write control unit reads the weight data required for the current convolution calculation into the global weight register array before calculation;

[0021] S103, starting the processing unit to perform a calculation task. During the calculation process, the weight mapping control unit maps the target part of the weight data stored in the global weight register array to the local mapping of the weight register of the current processing unit according to the current calculation state; at the same time, the weight read and write control unit reads the weight data required for the next convolution calculation into the cache of the global weight register array in advance;

[0022] S104: After completing the current calculation task, the weight mapping control unit updates the mapping and repeats step S103 to perform the next calculation task until the current convolution calculation is completed.

[0023] S105: Accumulate and activate the intermediate results of the calculation to obtain the final result of the convolution calculation;

[0024] S106: Start the next convolution calculation and repeat steps S103-S106 until all calculations are completed;

[0025] S107: Return to idle state.

[0026] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0027] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0028] The present invention also provides a processor, which is used to run a program, wherein the program executes the above method when it is run.

[0029] The above technical solution of the present invention has the following advantages over the prior art:

[0030] The neural network-enhanced graphics processor pipeline architecture of the present invention optimizes the GPU pipeline architecture for neural network computing acceleration scenarios, performs neural network-related enhancements on the traditional pipeline, expands the global weight register array dedicated to weights with high-speed access, and designs a specially optimized register mapping method to avoid the performance loss and power consumption overhead caused by repeated transfer of weights between computing components and shared memory or even global memory, thereby achieving a significant improvement in neural network computing performance at a small cost in area while maintaining graphics processing capabilities.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0033] Figure 1 This is a schematic diagram of a neural network-enhanced graphics processor pipeline architecture in Example 1 of the present invention;

[0034] Figure 2 Schematic diagram of a global weight register array in Embodiment 1 of the present invention;

[0035] Figure 3This is the mapping method for dividing the feature map of the same layer into different PEs for parallel processing in the first embodiment of the present invention;

[0036] Figure 4 This is the mapping method for dividing feature maps of different layers into different PEs for parallel processing in the first embodiment of the present invention;

[0037] Figure 5 This is a mapping method for dividing features at the same position of feature maps of different layers into different PEs for parallel processing in the first embodiment of the present invention;

[0038] Figure 6 This is a flowchart of a method for performing neural network calculations on a graphics processor in a second embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0040] Example 1

[0041] Reference Figure 1 As shown, this embodiment discloses a neural network enhanced graphics processor pipeline architecture, which includes:

[0042] A stream multiprocessor, the stream multiprocessor comprising at least one processing unit, an instruction fetch and decode unit, and a memory access control unit; all processing units in the stream multiprocessor share the instruction fetch and decode unit; all processing units in the stream multiprocessor share the memory access control unit and access a data bus under the control of the memory access control unit. The neural network enhanced graphics processor pipeline architecture further comprises:

[0043] A weight read and write control unit, a global weight register array and a weight mapping control unit, wherein the weight mapping control unit is provided in each processing unit, the weight read and write control unit is connected to the data bus, and all processing units in the stream multiprocessor share the weight read and write control unit and the global weight register array; wherein the data bus refers to the system bus used for data reading and writing in the GPU, and the instruction bus refers to the system bus used for instruction reading inside the GPU, and the two can be shared or independent buses depending on the design.

[0044] The weight read and write control unit is used to read the weight data required for the current convolution calculation into the global weight register array before the calculation; the weight mapping control unit is used to map the target part of the weight data stored in the global weight register array to the local mapping of the weight register of the current processing unit according to the current calculation status during the calculation process; the weight read and write control unit is also used to read the weight data required for the next convolution calculation into the cache of the global weight register array in advance.

[0045] Furthermore, the processing unit includes an instruction distribution unit, a data reading unit, an enhanced scalar data path and a write-back unit, which are connected in sequence. The instruction distribution units of all processing units share the instruction fetch and decoding unit. The instruction distribution unit is used to complete the distribution of instructions, and the weight mapping control unit is connected to the data reading unit.

[0046] Furthermore, the neural network enhanced graphics processor pipeline architecture also includes a global controller, and all processing units in the stream multiprocessor share the global controller and are uniformly controlled by the global controller.

[0047] Furthermore, the neural network enhanced graphics processor pipeline architecture also includes a shared memory unit, and all processing units in the streaming multiprocessor share the shared memory unit under the control of the memory access control unit.

[0048] The present invention takes into account the particularity of weights in neural network calculations. In order to better utilize the locality of weights, reduce the PE's access requirements to shared storage and improve the efficiency of weight data access, a global weight register array is added to the SM, which is responsible for storing the weight data required for calculation. In particular, the global weight register file is not accessed through memory access instructions, but is accessed in the form of registers. All PEs within an SM map the required part of the global weight register array to their own register space (i.e., local mapping of weight registers) through the weight mapping control unit therein to realize the use and sharing of the global weight register array.

[0049] To improve computational efficiency and reduce or even completely hide the time overhead of moving weights from system memory to the global weight register array, a dedicated weight read and write control unit is used. If the global weight register unit uses double or multiple buffers, the weights required for the next calculation can be moved from system memory to the global weight register array in advance while the PE is performing calculations.

[0050] In this embodiment, referring to Figure 2The global weight register array is a multi-bank memory, and each bank can store all the weight values of a filter. For example, the weight size is 3x3x16, and the data width of each bank is at least the bit width of the weight value. The entire weight register can store a fixed number of convolution kernels. When all the weight data of a convolution layer exceeds the storage capacity of the global weight register array, it can be processed in blocks based on the depth and number dimensions of the convolution kernel, and the final calculation result is obtained by accumulation after multiple calculations.

[0051] Among them, the convolution kernel refers to a two-dimensional matrix used in the convolution layer of the neural network to perform a convolution operation on the channel of the input eigenvalue.

[0052] A filter refers to a three-dimensional matrix used in the convolutional layer of a neural network to perform convolution operations on all channels of the input eigenvalues and obtain a channel of the output eigenvalue. The depth of the matrix is equal to the number of channels of the input eigenvalues, which represents the number of convolution kernels it contains.

[0053] In this embodiment, a specific weight value can be determined by its bank address, row address, and column address. However, in order to improve access efficiency, PE access to the global weight register array is implemented in a register manner rather than a memory access manner. Therefore, there are two problems: First, the encoding space for registers in GPU instructions is limited, and it is difficult to store the bank address, row address, and column address in the instruction encoding; Second, there is only one instruction fetch and decoding unit in each SM in the GPU, so the instructions are shared by all PEs in an SM, and the convolution algorithm requires multiple PEs to access different weight data when processing in parallel.

[0054] Therefore, an independent weight mapping control unit is designed in each PE. It fixes part of the bank address, row address, and column address according to the convolution parameters (such as the depth of the convolution) that the current PE is responsible for calculating, and decodes the other part of the address from the instruction, thereby achieving the effect of mapping the target part of the global weight register array to the register space of the PE.

[0055] The specific mapping method is related to the parallel mode used by the GPU for convolution calculation. When initializing, the weight mapping control unit should be configured to give the parallel mode and the length data of each dimension. Figure 3-5 , several examples of mapping methods corresponding to parallel schemes are given. The mapping methods for other parallel situations can also be easily obtained, so we will not go into details here.

[0056] The neural network-enhanced graphics processor pipeline architecture of the present invention optimizes the GPU pipeline architecture for neural network computing acceleration scenarios, performs neural network-related enhancements on the traditional pipeline, expands the global weight register array dedicated to weights with high-speed access, and designs a specially optimized register mapping method to avoid the performance loss and power consumption overhead caused by repeated transfer of weights between computing components and shared memory or even global memory, thereby achieving a significant improvement in neural network computing performance at a small cost in area while maintaining graphics processing capabilities.

[0057] Example 2

[0058] Reference Figure 6 This embodiment provides a method for performing neural network calculations on a graphics processor, which is applied to the neural network-enhanced graphics processor pipeline architecture in Example 1 and includes the following steps:

[0059] S101: Initialize and configure the parallel mode and hyperparameter information of the weight reading and writing control unit and the weight mapping control unit;

[0060] S102, the weight read and write control unit reads the weight data required for the current convolution calculation into the global weight register array before calculation;

[0061] S103, starting the processing unit to perform a calculation task. During the calculation process, the weight mapping control unit maps the target part of the weight data stored in the global weight register array to the local mapping of the weight register of the current processing unit according to the current calculation state; at the same time, the weight read and write control unit reads the weight data required for the next convolution calculation into the cache of the global weight register array in advance;

[0062] S104: After completing the current calculation task, the weight mapping control unit updates the mapping and repeats step S103 to perform the next calculation task until the current convolution calculation is completed.

[0063] S105: Accumulate and activate the intermediate results of the calculation to obtain the final result of the convolution calculation;

[0064] S106: Start the next convolution calculation and repeat steps S103-S106 until all calculations are completed;

[0065] S107: Return to idle state.

[0066] Example 3

[0067] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method in the above-mentioned embodiment 2 are implemented.

[0068] Example 4

[0069] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method in the above-mentioned embodiment 2 are implemented.

[0070] Example 5

[0071] This embodiment provides a processor, which is used to run a program, wherein the program executes the method in the above-mentioned embodiment 2 when running.

[0072] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0073] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0076] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications derived therefrom remain within the scope of protection of the present invention.

Claims

1. A neural network enhanced graphics processor pipeline system, comprising a streaming multiprocessor, wherein the streaming multiprocessor comprises at least one processing unit, an instruction fetch and decode unit, and a memory access control unit, wherein all processing units in the streaming multiprocessor share the instruction fetch and decode unit; all processing units in the streaming multiprocessor share the memory access control unit and access a data bus under the control of the memory access control unit, wherein: Also includes: A weight read and write control unit, a global weight register array, and a weight mapping control unit, wherein the weight mapping control unit is provided in each processing unit, the weight read and write control unit is connected to the data bus, and all processing units in the stream multiprocessor share the weight read and write control unit and the global weight register array; The weight read and write control unit is used to read the weight data required for the current convolution calculation into the global weight register array before the calculation; the weight mapping control unit is used to map the target part of the weight data stored in the global weight register array to the local mapping of the weight register of the current processing unit according to the current calculation status during the calculation process; the weight read and write control unit is also used to read the weight data required for the next convolution calculation into the cache of the global weight register array in advance.

2. The neural network enhanced graphics processor pipeline system according to claim 1, characterized in that The global weight register array is a multi-bank memory, and each bank can store the weight value of a filter.

3. The neural network enhanced graphics processor pipeline system according to claim 2, wherein: The specific weight value can be determined by its bank address, row address, and column address.

4. The neural network enhanced graphics processor pipeline system of claim 1, wherein: A global controller is also included. All processing units in the stream multiprocessor share the global controller and are uniformly controlled by the global controller.

5. The neural network enhanced graphics processor pipeline system according to claim 1, wherein: A shared storage unit is also included. All processing units in the streaming multiprocessor share the shared storage unit under the control of the memory access control unit.

6. The neural network enhanced graphics processor pipeline system of claim 1, wherein: The weight mapping control unit is used to map the target part of the global weight register to the register space of its own processing unit according to the current calculation state during the calculation process, including: The weight mapping control unit is used to fix partial values of the bank address, row address, and column address according to the convolution parameters calculated by the current processing unit during the calculation process, and decode another partial value of the address from the instruction, thereby mapping the target part of the global weight register array to the register space of the current processing unit.

7. A method for performing neural network calculations on a graphics processor, applied to the neural network enhanced graphics processor pipeline system according to claim 1, characterized in that: The following steps are involved: S101: Initialize and configure the parallel mode and hyperparameter information of the weight reading and writing control unit and the weight mapping control unit; S102, the weight read and write control unit reads the weight data required for the current convolution calculation into the global weight register array before calculation; S103, starting the processing unit to perform a calculation task. During the calculation process, the weight mapping control unit maps the target part of the weight data stored in the global weight register array to the local mapping of the weight register of the current processing unit according to the current calculation state; at the same time, the weight read and write control unit reads the weight data required for the next convolution calculation into the cache of the global weight register array in advance; S104: After completing the current calculation task, the weight mapping control unit updates the mapping and repeats step S103 to perform the next calculation task until the current convolution calculation is completed. S105: Accumulate and activate the intermediate results of the calculation to obtain the final result of the convolution calculation; S106: Start the next convolution calculation and repeat steps S103-S106 until all calculations are completed; S107: Return to idle state.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to claim 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to claim 7 are implemented.

10. A processor, characterized in that: The processor is configured to run a program, wherein the method according to claim 7 is executed when the program is run.

Images