Neural network processor based on SIMT and task execution method thereof

Abstract

The invention provides an SIMT-based neural network processor and a task execution method thereof, and the method comprises the steps that a general processor queries a state register of a coprocessor, and the state register stores the resource condition of the coprocessor; the universal processor completes splitting from a thread block to a thread bundle according to the resource condition, and the universal processor forwards a thread bundle instruction to a thread bundle distributor of the coprocessor; and the thread beam distributor decodes instructions in the thread beams in sequence and schedules the instructions to instruction queues of different calculation cores, and the thread beams sequentially perform thread beam scheduling, instruction emission and instruction execution according to the sequence, so that all calculations of a neural network task are completed in parallel, and a running result of the neural network task is obtained. According to the method, a general processor for task splitting is introduced in front of the thread beam scheduler or a specific compiler is directly used, so that dynamic scheduling of the threads can be realized, and a scheme for dynamically expanding the number of the threads according to data precision becomes a feasible architecture option.

Description

Technical Field

[0001] The present invention relates to the technical field of processor architecture for neural network training and reasoning in the field of machine learning, and in particular to a SIMT-based neural network processor and its task execution method, client, electronic device, computer-readable storage medium and computer program product. Background Art

[0002] In recent years, neural network algorithms have become the core of artificial intelligence research, thanks to their outstanding performance in fields such as computer vision, natural language processing, and speech recognition. However, with the continuous expansion of model size, especially the rise of large-scale models, their computational complexity and storage requirements have increased exponentially, placing increasing demands on hardware acceleration.

[0003] To address this challenge, at the hardware level, general-purpose graphics processing units (GPGPUs) employ a single-instruction, multiple-thread (SIMT) architecture, which divides high-level tasks into a large number of independent threads for parallel processing. This multi-threaded design is not only easy to program and highly flexible, but also demonstrates high efficiency and wide applicability in general-purpose parallel computing. In contrast, dedicated neural network processing units (NPUs) and tensor processing units (TPUs) have optimized their architectures based on the characteristics of neural network algorithms, focusing on efficiently executing different neural network tasks, thereby significantly improving performance and energy efficiency.

[0004] At the algorithmic level, quantization technology for neural network models has become a key tool for hardware acceleration. By converting floating-point calculations to low-precision integers, quantization significantly reduces computational complexity and storage requirements, significantly reducing hardware power consumption and improving computing performance. This not only improves hardware resource utilization but also provides important support for optimizing the energy efficiency of neural networks.

[0005] Among existing technologies, GPGPUs are designed for general-purpose computing and high-performance computing, rather than focusing on neural network training and inference. Therefore, in these scenarios, GPGPUs often struggle to fully realize their performance, especially in low-precision computing, where existing hardware architectures often suffer from low efficiency. In contrast, NPUs and TPUs are deeply optimized for neural network training and inference, achieving high computational efficiency at a variety of precisions, but their flexibility is limited.

[0006] Therefore, the core issue currently faced is how to design a hardware architecture that can take into account the flexibility of SIMT and the high efficiency of NPU and TPU. This architecture can not only flexibly adapt to different neural network models, but also efficiently support various quantized calculations.

[0007] The current GPGPU architecture has significant bottlenecks when processing quantitative calculations. The number of threads in the thread bundle (Warp) of the traditional SIMT architecture is fixed, with common configurations of 32, 64, or 128. This is because in the SIMT architecture design, the use of a fixed number of threads can simplify the design of the scheduler, compiler, and thread synchronization, without changing the core thread model, facilitating rapid implementation and achieving actual performance benefits. Over time, this early route has gradually solidified into the mainstream solution, and the related ecosystem (such as CUDA, compilers, and libraries) has also been built around a fixed thread model, making it very rare for the entire industry to explore similar solutions with dynamic changes in the number of threads for a long time. Although this fixed thread number design performs well in floating-point computing scenarios, it is difficult to fully tap the performance potential of low-precision computing units in quantitative computing, resulting in a significant drop in resource utilization.

[0008] To address this issue, existing designs attempt to improve the utilization of low-precision computing units through the following two approaches:

[0009] The first approach involves a hardware-based multi-issue mechanism, which activates more low-precision computational units by simultaneously launching multiple warps. However, this approach places extremely high demands on the hardware's instruction-issuing bandwidth. This significantly increases hardware complexity and power consumption, especially when there are a large number of low-precision computational units, making it difficult to implement on resource-constrained devices.

[0010] The second approach is to approach the problem from a software perspective, mapping the calculation of multiple output elements to a single thread through specific API encapsulation, so that the thread's computational requirements match the ratio of low-precision computing units. Although this approach alleviates the hardware bottleneck to a certain extent, it limits the flexibility of the SIMT architecture, reduces the independence between threads, and places an additional burden on programming and optimization.

[0011] While the two aforementioned solutions can improve computing unit utilization to a certain extent, they still cannot effectively address the issue of underutilized register resources caused by quantized computing, and they also lack sufficient flexibility and scalability. Further research has revealed that NPUs and TPUs optimize low-precision computing performance through customized data paths, but the hardware design of these specialized processors is relatively simple, relying entirely on programmers or compilers for task scheduling. While this design can significantly reduce hardware overhead, it also limits system flexibility to a certain extent.

[0012] Therefore, how to improve hardware efficiency while ensuring architectural flexibility has become a technical difficulty that needs to be urgently addressed in this field. Summary of the Invention

[0013] The technical problem solved by the present invention is the potential problem of insufficient resource utilization of GPGPU when performing low-precision calculations and the problem of limited flexibility of NPU and TPU. The present invention is mainly used in AI scenarios, and the need for synchronization between threads is small (traditional SIMT is a general-purpose computing device, while the present invention is mainly oriented to calculations in the field of AI). Therefore, by introducing a general-purpose processor for task splitting in front of the thread bundle scheduler or directly using a specific compiler, dynamic scheduling of threads can be achieved, making the solution of dynamically expanding the number of threads according to data precision a feasible architectural option. Therefore, the present invention can automatically adjust threads for each kernel and each precision without relying on multi-emission and special APIs. Different kernels can coexist with multiple Warp sizes, and more threads can be run in different computing components with higher parallelism in low-precision tasks, thereby improving computing resource utilization, throughput and scheduling flexibility, and effectively supplementing the existing technical routes.

[0014] In view of the shortcomings of the existing technology, the present invention proposes a neural network processor based on SIMT, which is characterized by comprising: a high-performance general-purpose processor, a neural network processing unit and a memory system;

[0015] This high-performance general-purpose processor is used to obtain instructions and data related to task division from the memory system, divide the neural network task into subtasks, and send the divided subtasks to the neural network processing unit in sequence; the tasks (neural network tasks) obtained from the memory system include: image semantic segmentation tasks, natural language processing tasks, speech recognition tasks, etc.

[0016] The neural network processing unit is configured to obtain computing instructions and data from the memory system to execute the subtask in parallel and obtain subtask calculation results. The neural network processing unit includes a thread block scheduler, y processing groups, and a secondary storage. The thread block scheduler is configured to schedule the subtask issued by the high-performance general-purpose processor and assign the subtask to a specific processor based on the resource availability of the processors in each processing group. The processing group is composed of x processors, and the x processors obtain instructions and data from the secondary storage to complete the neural network task calculation. The secondary storage is used for data exchange between the processor and the memory system, and for data interaction between the processing groups.

[0017] Each of the x processors consists of a streamlined processor, a coprocessor and a first-level storage; the streamlined processor accepts subtasks scheduled by the thread block scheduler, loads instructions and data related to the subtasks, and completes the division of subtasks and the calculation of scalar data; the streamlined processor sends relevant instructions to the coprocessor through the instruction forwarding unit; the coprocessor completes access to the first-level storage and performs neural network computing operations based on the received instructions, and integrates the results of all the computing operations as the task execution results of the image semantic segmentation task, the natural language processing task or the speech recognition task; the first-level storage is used to store data between the streamlined processor, the coprocessor and the second-level storage.

[0018] The SIMT-based neural network processor is characterized in that the coprocessor consists of a warp distributor and m computing cores; the warp distributor is used to package instructions sent by the streamlined processor and distribute them to the computing core, and the computing core performs neural network calculations according to the assigned tasks;

[0019] The warp dispatcher includes a decoder, a set of status registers, and a global scheduler. The decoder is used to decode instructions sent by the streamlined processor and send the decoding results to the status register and the global scheduler. The status register is used to store parameters related to coprocessor control and address. The global scheduler is used to schedule the decoded results of the decoder to specific computing cores.

[0020] The computing core includes an instruction queue, a warp scheduler, an instruction emission unit, a computing unit part, an interconnection unit and a register file; the instruction queue includes multiple queues, each queue stores the instructions of a warp, receives instructions scheduled by the global scheduler and caches them for scheduling by the warp scheduler; the warp scheduler is used to dispatch the warp instructions in each queue to the instruction emission unit for emission; the instruction emission unit is mainly responsible for the emission of instructions, and includes n scoreboards, one scoreboard for each warp; the computing unit part is used to execute instructions from the instruction emission unit and access the register file through the interconnection unit; the interconnection unit is mainly used to connect each computing unit with the registers in the register file; the register file is used to store computing-related data and allow each computing unit to exchange data;

[0021] The computing unit part includes V vector units, M matrix units and T tensor units, which respectively handle different operation granularities in neural network calculations; the engine in each computing unit has multiple data paths, including computing paths, memory access paths, data type conversion paths, and data movement paths; each data path is equipped with an independent or shared emission queue and an independent operand queue. The emission queue serves as an instruction scheduling window for out-of-order execution, and the operand queue serves as an operand buffer.

[0022] The SIMT-based neural network processor is characterized in that the high-performance general-purpose processor adjusts the number of threads in the thread bundle Warp of each computing core according to different computing precisions.

[0023] The present invention further proposes a task execution method for a neural network processor as claimed in claim 3, characterized by comprising:

[0024] Step 1: The general processor queries the status register of the coprocessor, which stores the resource status of the coprocessor. The general processor completes the splitting of the thread block into thread warps based on the resource status, and forwards the thread warp instruction to the thread warp distributor of the coprocessor.

[0025] In step 2, the thread bundle distributor decodes the instructions in the thread bundle in sequence and dispatches them to the instruction queues of different computing cores. The thread bundles are scheduled, issued, and executed in sequence, thereby completing all calculations of the neural network task in parallel and obtaining the running results of the neural network task.

[0026] The task execution method is characterized in that the resource situation is the number of warp threads supported by the current hardware; the general processor divides the thread block according to the number of warp threads to obtain multiple thread bundles Warp.

[0027] The task execution method is characterized in that the thread warp instruction is a SIMD instruction, which describes the same instruction that each thread in a warp needs to complete in lockstep.

[0028] The present invention also proposes a client for use with any one of the SIMT-based neural network processors described in claims 1-3.

[0029] The present invention also proposes an electronic device, characterized in that it includes a SIMT-based neural network processor as described in claims 1-3, and the electronic device is connected to an information display device, which is used to display the calculation operation results of the neural network using display parameters and attributes set by the user or through an artificial intelligence model.

[0030] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the task execution method of any of the neural network processors.

[0031] The present invention also proposes a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of the task execution method of any neural network processor.

[0032] It can be seen from the above scheme that the advantages of the present invention are:

[0033] The design of the present invention has high flexibility and computational efficiency when performing neural network training and inference tasks, and can effectively cover a variety of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the process of performing FP32 vector multiplication in a warp in traditional GPGPU;

[0035] Figure 2 Schematic diagram of the process of performing FP16 vector multiplication in a warp in traditional GPGPU;

[0036] Figure 3 For the present invention Figure 2 Schematic diagram of the process of vector multiplication after doubling the number of computing units and threads;

[0037] Figure 4 This is a hardware architecture diagram of the present invention;

[0038] Figure 5 Flowchart for executing neural network tasks of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that, in this application, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0040] Without further constraints, an element defined by the phrase "comprises a..." does not preclude the existence of additional identical elements in the process, method, article or apparatus that includes the element.

[0041] The processor of the present application is the control center of the electronic device, which can be one processor or a collective term of multiple processing elements. For example, it is one or more central processing units (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application, such as one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA).

[0042] Optionally, the processor can execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0043] In a specific implementation, as an embodiment, the processor can include one or more CPUs. Each of these processors can be a single-CPU or a multi-CPU. The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). The electronic device can include a server, a desktop computer, a notebook computer, a smartphone, a tablet computer, an embedded computer, etc., wherein the embedded computer includes vehicles and robots, etc.

[0044] The memory is used to store software programs for implementing the solutions of the present application, and is controlled by the processor to execute. The specific implementation can refer to the above method embodiments, which will not be repeated here.

[0045] It should be noted that the structure of the electronic device shown in the drawings of the present application does not constitute a limitation, and the actual knowledge structure recognition device can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0046] The above-described embodiments can be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0047] It should also be understood that the term "and / or" used herein is merely an association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it, but it can also represent an "and / or" relationship, which can be understood in the context before and after it.

[0048] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0049] It should also be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0050] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0051] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0052] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0053] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0054] To address these challenges, the present invention proposes a SIMT solution that supports a dynamic warp thread count. By optimizing the mapping between threads and hardware resources, the present invention overcomes the limitations of traditional warp designs, which typically use a fixed number of threads. The core of this solution is to dynamically adjust the number of threads in each warp, allowing it to flexibly change based on computational precision. Specifically, the lower the computational precision, the greater the number of threads in the warp, thereby fully utilizing low-precision computing units and significantly improving hardware resource utilization.

[0055] In addition, considering the wide range of neural network application scenarios and the diversity of device types, the present invention further designs a SIMT-based neural network processor architecture and its task execution method. By introducing multi-level computing modules, this architecture supports the expansion of hardware computing power in multiple dimensions, meeting the full range of scenario requirements from high-performance computing to lightweight edge computing. At the same time, through the online task partitioning mechanism, this architecture ensures the binary portability of program code, simplifies the programming and compiler development process, and reduces the workload of developers.

[0056] Through this series of innovative designs, the present invention not only solves the utilization problem of low-precision computing units, but also takes into account the flexibility, scalability and versatility of the architecture, ultimately providing an efficient, practical and widely adaptable solution for low-precision neural network calculations.

[0057] Specifically, in order to achieve the above technical effects, the present invention proposes the following key technical points:

[0058] Key point 1: A SIMT-based warp dynamic thread count design enables the SIMT architecture to use different numbers of threads to form warps at different computational precisions, thereby fully utilizing the hardware's own resources and further accelerating the computation of quantized neural networks.

[0059] Figure 4 The general purpose processor is a module used to dynamically allocate the number of Warp threads for different computational precisions. At the same time, this module provides binary portability for the system, that is, a code can run on systems of different hardware sizes under the same architecture without modifying the code or compiler, which enhances the compatibility of the system and simplifies development. The main functions of this module are shown in Figure 5 It is mainly used to split the kernel code into tasks, dynamically split it into multiple thread blocks, and provide it to the thread block scheduler for scheduling and subsequent execution.

[0060] Key point 2: A SIMT-based neural network processor architecture and its task execution method; it combines the flexibility of GPGPU with the efficiency of NPU and TPU, while supporting multi-dimensional computing power expansion and code binary portability. Figure 4 As can be seen in the figure, the expansion of computing power can be reflected in multiple dimensions. Specifically, the overall computing power of the system can be expanded by increasing the number of processing groups, increasing the number of processors in the processing group, increasing the number of computing cores in the coprocessor, and increasing the number of vector, matrix, and tensor computing units in the computing core; Binary portability of code: in Figure 4As can be seen in the processor, a general-purpose processor is added before the input of the thread warp distributor. This processor is used to dynamically split computing tasks and implement the construction of warps with different computing precisions. Therefore, the same code can run on systems of different hardware sizes under this architecture without recompilation or reconstruction, which supports seamless migration of binary executable files.

[0061] To illustrate the above-mentioned features and effects of the present invention more clearly and easily, the following embodiments are specifically described below with reference to the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are for illustrative purposes only. The scope of protection of the present invention is not limited to the disclosed embodiments; the present invention is defined by the appended claims.

[0062] In traditional GPGPU architectures, a warp typically contains a fixed number of threads, such as 32 threads, and each thread has multiple independent 32-bit registers and a computing unit.

[0063] Figure 1 This example demonstrates the process of executing a multiplication of FP32 vectors a and b to obtain FP64 vector c in a traditional GPGPU warp, representing a 32-precision calculation. In this example, R1, R2, R3, and R4 are four registers. R1 and R2 serve as source registers to store vectors a and b, while R3 and R4 serve as destination registers to store the low- and high-bit portions of vector c. This means that vector c has 32 elements, each consisting of two parts: the high- and low-bit portions, placed in two adjacent registers. Vectors a, b, and c each contain 32 elements, corresponding to the 32 threads in the warp. Specifically, at the logical level, thread 1 includes elements a1 (FP32), b1 (FP32), and c1 (FP64); at the physical level, thread 1 corresponds to register bits 0-31 and compute unit 1.

[0064] exist Figure 1 In the example, traditional GPGPU does not have efficiency problems because the precision required for calculation (the number of input data bits) just matches the width of the thread register, and resources are effectively utilized.

[0065] However, with the maturity of neural network quantization technology, modern neural network models increasingly use lower precision calculations, such as 4-bit, 8-bit, and 16-bit, to accelerate model inference and training. Figure 2The process of a Warp executing FP16 vector multiplication with FP16 vector to get FP32 vector in traditional GPGPU is shown. In this example, R1, R2 and R3 are three registers respectively, R1 and R2 store vector a and vector b as source registers, and R3 stores vector c as a destination register. Vector a, b and c also each contain 32 elements, corresponding to 32 threads in the Warp. At the logical level, thread 1 includes elements a1 (FP16), b1 (FP16) and c1 (FP32); at the physical level, thread 1 corresponds to the 0-31 bit fields of each register and computing unit 1.

[0066] Compared with Figure 1 Figure 2 In the traditional GPGPU architecture, the width of each element in the vector is changed from 32 bits to 16 bits, while the total number of elements stored in the register remains unchanged (32-element operation, one element per thread, a total of 32 threads), only the lower 16 bits of each register is in an unused state, resulting in resource waste. Figure 3 On the basis of Figure 2 , a doubled computing unit and thread (load) are added, so as to fully utilize the idle registers.

[0067] In the example of Figure 2 , the traditional GPGPU architecture has potential efficiency problems, especially in the case of low-precision calculation. This is because the bit width of the elements in vectors a and b is less than the width of the thread register, resulting in only a part of the storage space of each thread register being occupied during storage, thereby causing storage resource waste.

[0068] In addition, in order to accelerate the training and inference of quantized neural networks, existing designs usually configure more computing components for low-precision calculation, but how to fully utilize these additional computing resources has become a new challenge. For this purpose, there are mainly two ways in the existing solutions:

[0069] The first solution is to use a multi-launch mechanism at the hardware level, which activates more low-precision computing units by launching multiple Warps at the same time. This method can improve the utilization of computing resources, but it also puts higher requirements on the hardware, especially when the number of low-precision computing units is large, the hardware may need extremely high instruction launch bandwidth to make these computing units work efficiently.

[0070] ​The second solution is to use a specific application programming interface (API) to program the SIMT architecture at the software level. By encapsulating the calculation of multiple output elements in a thread, the number of elements processed by the threads in a warp matches the number of low-precision computing units. However, this approach reduces the flexibility of the SIMT architecture because it limits the independence between threads and may affect programming convenience.

[0071] To effectively address the above issues, the present invention proposes a solution for dynamic warp thread count. This solution allows the hardware to dynamically adjust the number of threads in each warp based on different computational precisions. This computational precision is specifically the bit width of the data to be calculated in the warp, which participates in the computation in the coprocessor hardware. Specifically, assuming the input computational precision is 64-bit floating-point or fixed-point, and the number of threads in the warp is t, then for other computational precisions, the mapping relationship between thread register width, number of warp threads, number of computing components, and register width can be shown in Table 1. Here, single register width = thread register width × number of threads.

[0072]

[0073] Table 1

[0074] There are two main approaches to dynamic threading: one is to use the compiler, and the other is to Figure 4 In the architecture shown, a general-purpose processor is added before the thread warp scheduler. The processor uses Strip mining technology to split the load according to the maximum Warp thread warp supported by the current hardware (different computing precisions).

[0075] The thread register width refers to the width of each register allocated to each thread in the register file; the number of warp threads is the maximum number of parallel running threads supported by the computing unit (vector, matrix, tensor) (the smallest scheduling unit); the number of computing components is the number of computing components equipped in the computing engine in the computing unit; the register width is the total width of each register in the register file, and its size = thread register width * number of warp threads.

[0076] Figure 3 Demonstrates GPGPU execution and performance after adopting the Warp dynamic thread number technology of the present invention Figure 2 The same FP16 vector is multiplied with the FP16 vector to get the FP32 vector. Figure 2 The configuration is different in Figure 3In the example, R1, R2, R3, and R4 are four registers. R1 and R2 serve as source registers to store vectors a and b, respectively, while R3 and R4 serve as destination registers to store the lower 16 bits and upper 16 bits of vector c. Vectors a, b, and c each contain 64 elements, corresponding to the 64 threads in a warp. Specifically, at the logical level, thread 1 includes three elements: a1 (FP16), b1 (FP16), and c1 (FP32). At the physical level, thread 1 includes bits 0 to 15 of each register and computation unit 1 in the computation unit.

[0077] Doubling the number of threads in a compute unit also doubles the number of compute components (see Table 1 above for details), so the number of vector, matrix, and tensor units does not need to change (the compute engine in each unit is equipped with a different number of compute components for different computational precisions to match the change in the number of warp threads).

[0078] exist Figure 3 In the example of Figure 2 The two methods are the same, but due to the increase in the number of threads in the Warp, on the one hand, the thread register width is reduced, and the register resources are more fully utilized; on the other hand, computing resources can be increased to improve the parallelism of low-precision calculations, thereby increasing the computing density per unit area of ​​the chip and significantly improving computing efficiency.

[0079] Figure 4 A neural network processor system based on the SIMT architecture was demonstrated, which is an example of Warp's dynamic thread count technology. The system includes a high-performance general-purpose processor, a neural network processing unit, and a memory system, which work together to complete neural network computing tasks.

[0080] The high-performance general-purpose processor CPU is mainly responsible for the division of high-level tasks and the management of task execution. The high-performance general-purpose processor obtains instructions and data related to task division from the memory system, and sends the divided subtasks to the neural network processing unit in an orderly manner according to the order specified in the program. The instructions related to task division refer to the task call and scheduling commands that control the execution of the neural network processing unit, and the data includes the input data, division parameters and scheduling configuration of each subtask. The neural network processing unit focuses on the parallel acceleration of neural network calculations. It obtains computing instructions and data (such as instructions and input data involved in subtasks, including but not limited to vector, matrix, tensor related instructions and data) from the memory system and completes specific neural network related calculations. The memory system provides global storage for the high-performance general-purpose processor and the neural network processing unit, and can be an independent or shared structure according to design requirements.

[0081] The neural network processing unit (NPU) consists of a thread block scheduler, y processing groups, and a secondary storage. The thread block scheduler is responsible for scheduling subtasks issued by high-performance general-purpose processors and allocating them to specific processors based on the processor resource availability of each processing group. A processing group consists of x processors, which retrieve instructions and data from the secondary storage to complete the calculation of the neural network task. The secondary storage is the largest memory within the NPU and is used for data exchange between the processors and the memory system. It also handles data exchange between processing groups. The secondary storage can be implemented as a cache or buffer. In some cases, a primary storage layer can be added within the processing group to enable data sharing between processors within the group.

[0082] Each processor primarily consists of a general-purpose processor (a streamlined CPU), a coprocessor, and primary storage. The general-purpose processor accepts subtasks scheduled by the thread block scheduler, loads relevant instructions and data, and completes the subtask division and scalar data calculations. The processor forwards relevant instructions to the coprocessor via the instruction forwarding unit. Based on the received instructions, the coprocessor accesses the primary storage and performs neural network computations. The primary storage is the largest memory within the processor and is responsible for storing data between the general-purpose processor, coprocessor, and secondary storage. It can be a cache or buffer, used for high-speed data exchange within the processor.

[0083] The coprocessor consists of a warp distributor and m compute cores. The warp distributor is responsible for packaging instructions sent by the general-purpose processor and assigning them to the compute cores. The compute cores perform specific neural network calculations based on the assigned tasks.

[0084] The warp dispatcher primarily consists of a decoder, a set of status registers, and a global scheduler. The decoder's primary function is to decode instructions sent by the general-purpose processor and send the decoded results to the status registers and the global scheduler. The status registers are used to store parameters related to coprocessor control and addresses, which can be modified and read by instructions. The relevant results are then transmitted back to the general-purpose processor or used in computations on the compute cores. The global scheduler is responsible for dispatching the decoded results to specific compute cores.

[0085] The computing core includes an instruction queue, a thread bundle scheduler, an instruction issue unit, a computing unit part, an interconnection unit, and a register file. The instruction queue includes a plurality of queues, each of which stores instructions of a warp, accepts instructions dispatched by a global scheduler, and buffers the instructions for dispatch by the thread bundle scheduler; the thread bundle scheduler is responsible for dispatching the warp instructions in the queues to the instruction issue unit for transmission; the instruction issue unit is mainly responsible for the transmission of instructions, includes n scoreboards, each for a warp, and includes a register renaming module for renaming the same-named registers used in different warps. There are mainly two cases: 1. There are pseudo-data correlations (read-after-write and write-after-write) between registers (operands) used in different warps; 2. The system supports runtime task splitting (binary portability), so when writing code, each thread uses the same logical register (because when writing code, it is unknown how each thread will be scheduled for execution in hardware), so the runtime needs to rename these logically identical registers to different registers to remove this pseudo correlation.

[0086] The computing unit part executes instructions from the instruction issue unit without conflicts and accesses the register file through the interconnection unit; the interconnection unit is mainly used to connect the registers in the register file and the computing units; the register file is used to store computing-related data and allows data interaction between the computing units, and can adopt a shared or separated structure.

[0087] The computing unit part includes V vector units, M matrix units, and T tensor units, which process different operation granularities in neural network computing, cooperate with the scalar operation capability of the general processor, and can fully meet the computing needs in the neural network training and inference process. In addition, the engines in each computing unit can include a variety of data paths, including computing paths, memory access paths, data type conversion paths, data movement paths, etc. Each data path can be equipped with independent or shared issue queues and independent operand queues. The issue queue serves as a dispatch window for out-of-order execution instructions, and the operand queue serves as an operand buffer to solve the memory access conflict of the register file.

[0088] The above scalar is relative to the concepts of vector, matrix, and tensor, that is, single instruction single data (SISD), such as ALU operation.

[0089] To further explain the workflow of the present application, Figure 5The main execution process of the task in the application is shown. Before the task starts, the high-performance general processor divides the kernel of the high-level task of the neural network inference task into multiple thread blocks according to the programmer's intention through the parallel programming framework, and stores the thread blocks and the data parameters of the neural network in the memory system. The high level refers to a higher level than the kernel level, such as a kernel set composed of a certain neural network inference operator. After the task starts, the high-performance general processor sends the parameters of each thread block to the thread block scheduler in a specific order according to the high-level task control instruction. Then, the thread block scheduler schedules the parameters of the thread block to the specific general processor according to the resource situation of each processor.

[0090] When the general processor receives the thread block parameters, it loads the specific code of the thread block from the memory system through each level of storage, and executes the code segment to realize the division of the thread block into a thread bundle in line. Specifically, first, the general processor queries the state register of the coprocessor, which stores the resource situation of the coprocessor. Then, the general processor completes the division of the thread block into a thread bundle according to the feedback state register value, using the Warp dynamic thread bundle technology. Finally, the general processor forwards the thread bundle instruction to the thread bundle allocator of the coprocessor.

[0091] The feedback state register value is mainly the number of Warp threads currently supported by the hardware. According to this value, the general processor can use the Strip mining technology to divide the thread block according to the number of Warp threads to obtain multiple thread bundles (Warp). The thread bundle instruction is a SIMD instruction, which describes the same instruction that each thread in a Warp wants to lock in step, mainly composed of configuration, calculation, memory access and other instructions. Lock step means strict synchronization, that is, each thread can operate on different data, but all threads execute the same operation at each clock cycle.

[0092] Further, the thread bundle allocator decodes the instructions in the thread bundle in turn and schedules them to the instruction queue of different calculation cores. Finally, the thread bundle is sequentially scheduled, the instructions are emitted and executed, and all the calculations of the task are completed in parallel.

[0093] The following are specific system and method embodiments corresponding to the above-mentioned embodiments. The above-mentioned embodiments can be implemented in cooperation with the above-mentioned embodiments. The related technical details mentioned in the above-mentioned embodiments are still valid in this embodiment. In order to reduce repetition, they will not be repeated here. Correspondingly, the related technical details mentioned in this embodiment can also be applied in the above-mentioned embodiments.

[0094] The application provides a neural network processor based on SIMT, characterized by comprising a high-performance general processor, a neural network processing unit and a memory system.

[0095] This high-performance general-purpose processor is used to obtain instructions and data related to task division from the memory system, divide the neural network task into subtasks, and send the divided subtasks to the neural network processing unit in sequence; the tasks obtained from the memory system include: image semantic segmentation tasks, natural language processing tasks, speech recognition tasks, etc.

[0096] The neural network processing unit is configured to obtain computing instructions and data from the memory system to execute the subtask in parallel and obtain subtask calculation results. The neural network processing unit includes a thread block scheduler, y processing groups, and a secondary storage. The thread block scheduler is configured to schedule the subtask issued by the high-performance general-purpose processor and assign the subtask to a specific processor based on the resource availability of the processors in each processing group. The processing group is composed of x processors, and the x processors obtain instructions and data from the secondary storage to complete the neural network task calculation. The secondary storage is used for data exchange between the processor and the memory system, and for data interaction between the processing groups.

[0097] Each of the x processors consists of a streamlined processor, a coprocessor and a first-level storage; the streamlined processor accepts subtasks scheduled by the thread block scheduler, loads instructions and data related to the subtasks, and completes the division of subtasks and the calculation of scalar data; the streamlined processor sends relevant instructions to the coprocessor through the instruction forwarding unit; the coprocessor completes access to the first-level storage and performs neural network computing operations based on the received instructions, and integrates the results of all the computing operations as the task execution results of the image semantic segmentation task, the natural language processing task or the speech recognition task; the first-level storage is used to store data between the streamlined processor, the coprocessor and the second-level storage.

[0098] The SIMT-based neural network processor is characterized in that the coprocessor consists of a warp distributor and m computing cores; the warp distributor is used to package instructions sent by the streamlined processor and distribute them to the computing core, and the computing core performs neural network calculations according to the assigned tasks;

[0099] The warp dispatcher includes a decoder, a set of status registers, and a global scheduler. The decoder is used to decode instructions sent by the streamlined processor and send the decoding results to the status register and the global scheduler. The status register is used to store parameters related to coprocessor control and address. The global scheduler is used to schedule the decoded results of the decoder to specific computing cores.

[0100] The computing core includes an instruction queue, a warp scheduler, an instruction emission unit, a computing unit part, an interconnection unit and a register file; the instruction queue includes multiple queues, each queue stores the instructions of a warp, receives instructions scheduled by the global scheduler and caches them for scheduling by the warp scheduler; the warp scheduler is used to dispatch the warp instructions in each queue to the instruction emission unit for emission; the instruction emission unit is mainly responsible for the emission of instructions, and includes n scoreboards, one scoreboard for each warp; the computing unit part is used to execute instructions from the instruction emission unit and access the register file through the interconnection unit; the interconnection unit is mainly used to connect each computing unit with the registers in the register file; the register file is used to store computing-related data and allow each computing unit to exchange data;

[0101] The computing unit part includes V vector units, M matrix units and T tensor units, which respectively handle different operation granularities in neural network calculations; the engine in each computing unit has multiple data paths, including computing paths, memory access paths, data type conversion paths, and data movement paths; each data path is equipped with an independent or shared emission queue and an independent operand queue. The emission queue serves as an instruction scheduling window for out-of-order execution, and the operand queue serves as an operand buffer.

[0102] The SIMT-based neural network processor is characterized in that the high-performance general-purpose processor adjusts the number of threads in the thread bundle Warp of each computing core according to different computing precisions.

[0103] The present invention further proposes a task execution method for a neural network processor as claimed in claim 3, characterized by comprising:

[0104] Step 1: The general processor queries the status register of the coprocessor, which stores the resource status of the coprocessor. The general processor completes the splitting of the thread block into thread warps based on the resource status, and forwards the thread warp instruction to the thread warp distributor of the coprocessor.

[0105] In step 2, the thread bundle distributor decodes the instructions in the thread bundle in sequence and dispatches them to the instruction queues of different computing cores. The thread bundles are scheduled, issued, and executed in sequence, thereby completing all calculations of the neural network task in parallel and obtaining the running results of the neural network task.

[0106] The task execution method is characterized in that the resource situation is the number of warp threads supported by the current hardware; the general processor divides the thread block according to the number of warp threads to obtain multiple thread bundles Warp.

[0107] The task execution method is characterized in that the thread bundle instruction is a SIMD instruction, and the same instruction to be completed by the threads in the warp is described.

[0108] The application further provides a client for the SIMT-based neural network processor in any of claims 1-3.

[0109] The application further provides an electronic device, characterized in that the electronic device comprises the SIMT-based neural network processor in any of claims 1-3, and the electronic device is connected with an information display device, which is used to display the calculation operation result of the neural network according to the display parameters, attributes set by a user or through an artificial intelligence model.

[0110] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is used to implement the steps of the task execution method of any of the neural network processors when executed by a processor.

[0111] The application further provides a computer program product, which comprises a computer program, and the computer program is used to implement the steps of the task execution method of any of the neural network processors when executed by a processor.

Claims

1. A neural network processor based on SIMT, characterized in that: include: High-performance general-purpose processors, neural network processing units, and memory systems; The high-performance general-purpose processor is used to obtain instructions and data related to task division from the memory system, divide the neural network task into subtasks, and send the divided subtasks to the neural network processing unit in sequence; The neural network processing unit is used to obtain computing instructions and data from the memory system to execute the subtask in parallel and obtain the subtask computing results; the neural network processing unit includes a thread block scheduler, y processing groups and a secondary storage; The thread block scheduler is used to schedule the subtask issued by the high-performance general-purpose processor and assign the subtask to a specific processor based on the resource status of the processors in each processing group. The processing group is composed of x processors, and the x processors obtain instructions and data from the secondary storage to complete the neural network task calculation. The secondary storage is used for data exchange between the processors and the memory system, and for data interaction between processing groups. Each of the x processors consists of a streamlined processor, a coprocessor, and a primary storage; the streamlined processor accepts subtasks scheduled by the thread block scheduler, loads instructions and data related to the subtasks, and completes the division of subtasks and the calculation of scalar data; the streamlined processor sends relevant instructions to the coprocessor through an instruction forwarding unit; the coprocessor completes access to the primary storage and performs neural network computing operations based on the received instructions; the primary storage is used to store data between the streamlined processor, the coprocessor, and the secondary storage.

2. The SIMT-based neural network processor according to claim 1, wherein: The coprocessor consists of a warp distributor and m computing cores; the warp distributor is used to package instructions sent by the streamlined processor and distribute them to the computing core, which performs neural network calculations according to the assigned tasks; The warp dispatcher includes a decoder, a set of status registers, and a global scheduler. The decoder is used to decode instructions sent by the streamlined processor and send the decoding results to the status register and the global scheduler. The status register is used to store parameters related to coprocessor control and address. The global scheduler is used to schedule the decoded results of the decoder to specific computing cores. The computing core includes an instruction queue, a warp scheduler, an instruction emission unit, a computing unit part, an interconnection unit and a register file; the instruction queue includes multiple queues, each queue stores the instructions of a warp, receives instructions scheduled by the global scheduler and caches them for scheduling by the warp scheduler; the warp scheduler is used to dispatch the warp instructions in each queue to the instruction emission unit for emission; the instruction emission unit is mainly responsible for the emission of instructions, and includes n scoreboards, one scoreboard for each warp; the computing unit part is used to execute instructions from the instruction emission unit and access the register file through the interconnection unit; the interconnection unit is mainly used to connect each computing unit with the registers in the register file; the register file is used to store computing-related data and allow each computing unit to exchange data; The computing unit part includes V vector units, M matrix units and T tensor units, which respectively handle different operation granularities in neural network calculations; the engine in each computing unit has multiple data paths, including computing paths, memory access paths, data type conversion paths, and data movement paths; each data path is equipped with an independent or shared emission queue and an independent operand queue. The emission queue serves as an instruction scheduling window for out-of-order execution, and the operand queue serves as an operand buffer.

3. The SIMT-based neural network processor according to claim 2, wherein: The high-performance general-purpose processor adjusts the number of threads in a thread bundle Warp of each computing core according to different computing precisions.

4. A task execution method for a neural network processor as claimed in claim 3, characterized in that: include: Step 1: The general processor queries the status register of the coprocessor, which stores the resource status of the coprocessor. The general processor completes the splitting of the thread block into thread warps based on the resource status, and forwards the thread warp instruction to the thread warp distributor of the coprocessor. In step 2, the thread bundle distributor decodes the instructions in the thread bundle in sequence and dispatches them to the instruction queues of different computing cores. The thread bundles are scheduled, issued, and executed in sequence, thereby completing all calculations of the neural network task in parallel and obtaining the running results of the neural network task.

5. The task execution method according to claim 4, wherein: The resource status is the number of warp threads supported by the current hardware; the general-purpose processor divides the thread block according to the number of warp threads to obtain multiple thread bundles Warp.

6. The task execution method according to claim 4, wherein: The thread warp instruction is a SIMD instruction, which describes the same instruction that each thread in a warp must complete in lockstep.

7. A client for use with any one of the SIMT-based neural network processors described in claims 1-3.

8. An electronic device, characterized in that: Including a SIMT-based neural network processor as described in claims 1-3, the electronic device may be connected to an information display device, and the information display device is used to display the calculation operation results of the neural network using display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the task execution method of the neural network processor described in any one of claims 4 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the task execution method of the neural network processor described in any one of claims 4 to 6 are implemented.

Images