Configurable heterogeneous AI processor

Abstract

The embodiment of the present invention provides a configurable heterogeneous artificial intelligence processor, including at least two computing units of different structural types and a task queue, storage unit and controller corresponding to each computing unit; wherein the controller decomposes the computational graph of the neural network to be processed into multiple computing subtasks and distributes them to the corresponding task queues of each computing unit, sets the dependencies between the computing subtasks, and realizes synchronization between the computing subtasks and controls the access of the data involved in each computing subtask on the storage unit and the off-chip memory according to the set dependencies. The technical solution of the embodiment of the present invention adopts the form of on-chip heterogeneity, and a single controller uniformly schedules and manages the computing units of each architecture to process different application tasks, so that the artificial intelligence processor can flexibly adapt to different application scenarios, which not only improves the scalability, but also improves the efficiency of processing different tasks.

Description

Technical Field

[0001] The present invention relates to heterogeneous computing and processor architecture, and in particular to a configurable heterogeneous processor for artificial intelligence processing. Background Art

[0002] Artificial intelligence (AI) technology has developed rapidly in recent years and has penetrated into various fields such as visual perception, speech recognition, assisted driving, smart home, traffic dispatch, etc. Many AI algorithms involve learning and computing based on neural networks, such as convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), etc. These AI algorithms usually require strong parallel computing capabilities to process massive amounts of data. However, the central processing unit (CPU) or DSP, which executes serial instructions, is very inefficient in executing AI algorithms. As a result, a heterogeneous computing architecture of "CPU+accelerator chip" has emerged, in which AI chips are used to handle a large number of computing tasks in AI applications, while other computing tasks and scheduling tasks that cannot be undertaken are still handled by the CPU, such as processing irregular data structures, unpredictable access patterns, recursive algorithms, branch-intensive codes, and single-threaded programs; and AI chips are used as coprocessors, for example, to process tensor computing tasks related to AI algorithms. The disadvantages of this heterogeneity are that the CPU bottleneck makes it impossible to fully utilize the performance of the computing acceleration chip, so a multi-core CPU has to be used to remedy system performance, which brings unnecessary power consumption and cost; secondly, it adopts traditional heterogeneity, which is driven by multiple different software and has low coordination efficiency.

[0003] These dedicated chips for AI applications can also be called AI chips, artificial intelligence processors, coprocessors, AI accelerators, etc. At present, the main structures of AI chips can generally be divided into three categories: GPU, FPGA, and ASIC. Graphics Processing Unit (GPU) was first used for AI computing due to its good matrix computing capabilities and parallel computing advantages, but the chip size and power consumption are large, making it difficult to effectively improve performance. Field-Programmable Gate Array (FPGA) can be hardware programmed, but its performance is lower than that of GPU. Application Specific Integrated Circuits (ASIC) is a chip that can be customized for specific application requirements. It is smaller in size and has lower power consumption, but it has no programmability and poor adaptability and scalability. With the popularization of artificial intelligence algorithms in various application fields, there is an urgent need for a scalable and efficient artificial intelligence processor that can adapt to specific application requirements. Summary of the invention

[0004] Therefore, the purpose of an embodiment of the present invention is to provide a new configurable heterogeneous artificial intelligence processor, in which a controller uniformly manages and schedules different computing units in heterogeneous forms within the chip to process different artificial intelligence application tasks, thereby flexibly adapting to different application scenarios while also improving the execution efficiency of computing tasks.

[0005] The above purpose is achieved through the following technical solutions:

[0006] According to a first aspect of an embodiment of the present invention, there is provided a configurable heterogeneous artificial intelligence processor, comprising computing units of at least two different structural types and a task queue, a storage unit and a controller corresponding to each computing unit; the controller comprises a task scheduling module, a task synchronization module and a memory access control module; wherein the task scheduling module is used to decompose the computational graph of a neural network to be processed into a plurality of computing subtasks according to a configuration option indicating task allocation and distribute the decompositions to the task queues of the corresponding hardware of each computing unit, and to set the dependency relationship between the computing subtasks; the task synchronization module realizes the synchronization between the computing subtasks on the hardware according to the set dependency relationship; and the memory access control module is used to control the access of the data involved in each computing subtask on the storage unit and the off-chip memory.

[0007] In some embodiments of the present invention, the task scheduling module may also be configured to set the working mode between the computing units according to the configuration option indicating the working mode, wherein the working mode includes an independent parallel mode, a cooperative parallel mode and an interactive cooperative mode;

[0008] In independent parallel mode, the computational subtasks of each computing unit are executed independently and in parallel;

[0009] In cooperative parallel mode, the computational subtasks of each computing unit are executed cooperatively in a pipeline manner;

[0010] In the interactive cooperation mode, when executing a computing subtask, a computing unit needs to wait for or rely on the results generated by the computing subtask executed by the computing unit that cooperates with it.

[0011] In some embodiments of the present invention, the storage unit may include a cache memory and a scratchpad memory.

[0012] In some embodiments of the present invention, the memory access control module can also be configured to set the storage location of shared data between different computing units according to the working mode between the computing units, wherein: in independent parallel mode, the storage location is set on the off-chip memory; in cooperative parallel mode, the storage location is set on the note-type memory; in interactive cooperative mode, the storage location is set on the cache memory.

[0013] In some embodiments of the present invention, the task scheduling module is further configured to perform operation fusion on computing subtasks assigned to the same computing unit according to a configuration option indicating operation fusion.

[0014] In some embodiments of the present invention, the task scheduling module may also be configured to notify the memory access control module to save the output of the intermediate neural network layer to a scratchpad memory according to a configuration option indicating inter-layer fusion.

[0015] In some embodiments of the present invention, the structure type of the computing unit is one of the following: application-specific integrated circuit ASIC, general purpose graphics processing unit GPGPU, field programmable gate array FPGA, video codec DSP.

[0016] In some embodiments of the present invention, one of the at least two computing units of different structural types is a customized computing unit for a specific artificial intelligence algorithm or operation, and the other is a programmable computing unit.

[0017] In some embodiments of the present invention, the at least two computing units of different structural types may include a computing unit using an application specific integrated circuit ASIC structure and a computing unit using a general purpose graphics processing unit GPGPU structure.

[0018] In some embodiments of the present invention, the task scheduler is further configured to distribute computing subtasks to corresponding computing units according to the capabilities of each computing unit.

[0019] In some embodiments of the present invention, the memory access control module can be used to read the operation data required by each computing unit to perform the computing subtask from the off-chip memory to the storage unit through the memory access interface, and store the data generated by each computing unit after performing the computing subtask in the storage unit and / or the off-chip memory. The memory access control module may include a direct memory access (DMA) component to be responsible for data access and data transfer.

[0020] According to a second aspect of an embodiment of the present invention, there is provided an on-chip heterogeneous artificial intelligence processor, comprising: a plurality of computing clusters connected via an on-chip data exchange network, wherein each computing cluster comprises: computing units of at least two different structural types; a task queue corresponding to each computing unit, for storing computing subtasks to be executed by the computing unit; a memory access control module, and a cache and on-chip memory shared by each computing unit. The on-chip heterogeneous artificial intelligence processor also includes a controller, which is used to decompose the received computational graph of the neural network to be processed into a plurality of computing subtasks and distribute them to the corresponding command queues of the computing units in each computing cluster. The on-chip heterogeneous artificial intelligence processor also includes a memory access interface for accessing off-chip memory and a host interface for interacting with an off-chip main processor.

[0021] In some embodiments of the present invention, one of the at least two computing units of different structural types is a customized computing unit for a specific artificial intelligence algorithm or operation, and the other is a programmable computing unit.

[0022] In some embodiments of the present invention, the at least two computing units of different structural types include a computing unit adopting an application specific integrated circuit ASIC structure and a computing unit adopting a general purpose graphics processing unit GPGPU structure.

[0023] The technical solution of the embodiment of the present invention may have the following beneficial effects:

[0024] In the form of on-chip heterogeneity, a single controller uniformly schedules and manages computing units of various architectures to handle different application tasks, and can flexibly configure the working mode between computing units, so that the artificial intelligence processor can flexibly adapt to different application scenarios, which not only improves scalability, but also improves the efficiency of processing different tasks. This on-chip heterogeneous processor can support not only ordinary computing applications, but also a large number of parallel computing applications and artificial intelligence applications with deep neural network learning as the core.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. Obviously, the accompanying drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0027] In the attached picture:

[0028] Figure 1 A schematic diagram of the structure of an on-chip heterogeneous artificial intelligence processor according to an embodiment of the present invention is shown.

[0029] Figure 2 a), b) and c) are schematic diagrams showing working modes between different computing units according to an embodiment of the present invention.

[0030] FIG3(a) and (b) show a schematic diagram of a task scheduling and task synchronization process according to an embodiment of the present invention.

[0031] Figure 4 A schematic diagram of the structure of an on-chip heterogeneous artificial intelligence processor according to yet another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments in conjunction with the accompanying drawings. It should be understood that the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0033] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present invention. However, those skilled in the art will appreciate that the technical solution of the present invention can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present invention.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0036] As mentioned above, heterogeneous computing mainly refers to the way of computing using a system composed of computing units with different types of instruction sets and architectures, including the main processor (such as CPU) and other types of coprocessors, such as GPU, DSP, ASIC, FPGA, etc. Systems composed of heterogeneous computing units usually take the form of System on Chip (SoC). Although different types of computing units can be used to process different tasks to improve the overall utilization of computing resources, the scheduling and data exchange overhead is relatively large, which leads to a significant decrease in the operating efficiency of the entire system.

[0037] This is because, in terms of form, the system on chip (SoC) integrates IP functional modules such as processor modules (including CPU, GPU), memory modules, various interface control modules, and various interconnection buses on a single chip, but each IP module is independent, and each contains a complete hardware system and the embedded software it carries. This means that each IP module on the SoC chip is still driven by its own corresponding software, which leads to a relatively large scale of SoC hardware and software. Moreover, the SoC must also be equipped with standard protocol software for interconnection between IP functional modules from various IP suppliers, and its scheduling and data exchange overhead is relatively large. Moreover, the increasing variety and complexity of IP functional modules and the lack of common interfaces make the integration of IP increasingly difficult. In addition, this model also has the problem of data coherence. This is because in order for each IP functional module to perform parallel or asynchronous computing, each IP functional module is usually required to retain its own copy of the data, and the IP functional modules often need to share data. For example, one IP functional module may rely on the data generated by another IP functional module. This involves a complex shared cache mechanism or synchronization mechanism to ensure correct data exchange. At the same time, performance is degraded because the IP functional modules need to wait for each other.

[0038] In addition, in essence, the AI ​​chip used as the coprocessor itself in the above system is isomorphic. As mentioned in the background technology section, the three mainstream AI chip processor architectures GPU, FPGA, and ASIC each have their own advantages and disadvantages. GPU has good parallel computing capabilities, but it is specifically designed for image computing and is not specifically designed for AI algorithms. It is difficult to adapt to the AI ​​algorithm requirements of different application scenarios. Field programmable gate array FPGA can be programmed in hardware using languages ​​such as OpenCL, which improves the flexibility of the application and is suitable for multi-instruction, single data stream analysis, but has weak large-scale parallel computing capabilities such as model training. Application-specific integrated circuit ASIC is a customized chip for specific AI application requirements. It is smaller in size and has lower power consumption, but it cannot be programmed and has poor adaptability and scalability. With the increasing popularity of artificial intelligence algorithms in the cloud and in a variety of smart terminal devices, there is an urgent need for an scalable, more comprehensive and efficient artificial intelligence processor that can adapt to specific application requirements.

[0039] In an embodiment of the present invention, an artificial intelligence processor in a heterogeneous form on a chip is provided, which includes at least two computing units of architecture. For example, one of the architectures can use a programmable image processing unit (also called GPGPU) that can handle various general computing tasks, and the other architecture can use a dedicated integrated circuit ASIC customized for a specific artificial intelligence algorithm. Since a large number of matrix and derivative operations are involved in many AI algorithms, ASICs dedicated to processing matrix addition, multiplication and division, as well as ASICs for vector operations and derivative operations can be customized. In some embodiments, ASICs for various types of neural networks can also be customized, such as ASICs customized for CNN, ASICs customized for DNN, and ASICs customized for RNN. In some embodiments, ASICs can also be customized for various machine learning algorithms, such as ASICs customized for support vector machine algorithms, ASICs customized for K-means algorithms, and so on. Using these customized computing units for AI algorithms or AI operations can save power consumption and reduce chip area. For computing tasks that cannot be processed by these customized ASIC computing units, they are assigned to computing units of programmable GPGPU architecture for execution. In this way, not only can the execution efficiency of different computing tasks in artificial intelligence applications be improved, but also the needs of different application scenarios can be adapted, with good scalability. In some other embodiments, in addition to the above two architectures, the on-chip heterogeneous artificial intelligence processor may also include a computing unit in the form of an FPGA, a computing unit in the form of a video codec DSP, and even a general-purpose central processing unit CPU as one of the computing units. In the solution of this embodiment, various heterogeneous elements are organically integrated on a chip and seamlessly scheduled by a single controller, which not only gives full play to the unique advantages of each architecture, but also has no obvious bottlenecks, making AI computing more efficient.

[0040] The following is a more detailed introduction to the on-chip heterogeneous artificial intelligence processor of the embodiment of the present invention in conjunction with the accompanying drawings. For the convenience of description, the following only takes the processing of the neural network as the basis of the AI ​​algorithm as an example, and takes the ASIC computing unit customized for matrix operations (hereinafter also referred to as TU) and the programmable general-purpose graphics processing unit GPGPU (hereinafter also referred to as CU) responsible for the remaining computing tasks as examples of two different structural types of computing units for illustration, but does not impose any limitation on this.

[0041] Figure 1An on-chip heterogeneous artificial intelligence processor according to an embodiment of the present invention is provided, which includes at least two computing units of different structural types, for example, the first type of computing unit is TU, the second type of computing unit is CU, and a task queue corresponding to each computing unit, the task queue is used to store the computing subtasks that need to be executed by the computing unit. The on-chip heterogeneous artificial intelligence processor also includes a controller, a storage unit, and a memory access interface for accessing an off-chip memory such as DDR. The controller can be used to decompose the received computational graph of the neural network to be processed into multiple computing subtasks and distribute them to the corresponding task queues of each computing unit; the storage unit can be used to store the data required for each computing unit to execute the computing subtask. In some embodiments, the storage unit may include a cache memory such as L1 cache, L2 cache, and an on-chip memory such as a scratch-pad memory (SPM). Each computing unit can share the on-chip memory and cache memory.

[0042] exist Figure 1 In the artificial intelligence processor shown, the controller is mainly used to assign computing subtasks to each computing unit, control the synchronization between each computing subtask, and be responsible for the handling and control of the data involved in each computing subtask. It may include a task scheduling module, a task synchronization module, and a memory access control module. The task scheduling module can analyze the computing graph corresponding to the neural network program to be processed according to the characteristics of each current computing unit, decompose it into multiple computing subtasks, and distribute each computing subtask to the corresponding task queue of the computing unit that can process the computing subtask of this type. As mentioned above, computing units of different structural types can be used to process different tasks or instructions. For example, CPU is suitable for logical operations and branch-intensive control processes, GPU is suitable for high-throughput data processing and parallel computing, and ASIC-structured computing units customized for matrix operations and convolution calculations in deep learning are suitable for processing operations such as matrices and convolutions. DSP is suitable for real-time signal processing, video encoding and decoding, etc., and GPGPU-structured computing units can process user-defined operations and operations that cannot be performed by other TUs. Therefore, assigning the same type of computing subtasks to computing units suitable for processing this type of task as much as possible can improve execution efficiency and avoid internal switching overhead. In addition to the characteristics of different computing units, the task scheduling module can also allocate computing subtasks based on the processing capabilities of each computing unit. For example, it can distribute computing subtasks as evenly as possible among the computing units based on the current queuing status in the task queue of each computing unit, the current load of each computing unit, etc.

[0043] In some embodiments, the task scheduling module can also determine the dependency between the computing subtasks assigned to each computing unit according to the dependency between the nodes on the computing graph, and define synchronization event points in each task queue according to the corresponding dependency. The task synchronization module monitors the synchronization event points in each task queue to realize automatic synchronization between each computing subtask. In some embodiments, the synchronization event point can be a synchronization control flag. When the task scheduling module assigns the computing subtask, it determines whether there is a dependency between the current subtask and the subtask previously assigned to each task queue. If there is, the synchronization control flag is set for the current subtask. The synchronization control flag contains parameters for the computing subtask on which the current subtask depends, such as a task queue identifier and a task sequence number. The task synchronization module can monitor the computing subtasks indicated by each synchronization control flag. When it is detected that the computing subtask has been executed, the task synchronization module can send an event notification to the relevant task queue or record the execution status of the computing subtask for subsequent query. In this way, when the current subtask with a synchronization control flag is extracted from the task queue for execution, the task synchronization module can be queried first to see whether the dependent subtask has been executed. If not, the current subtask needs to wait.

[0044] For example, taking the processing of some operations in a neural network as an example, in a neural network, element-wise (EW) operations are suitable for CU computing units, while convolution is more suitable for TU computing units. In the computational graph of the neural network, there is often a need for sequential calculation of convolution + EW + convolution + EW. When performing task scheduling, the controller can assign all convolution tasks to TU and all EW tasks to CU, as long as the event identifiers for mutual waiting for synchronization are added to the task queue. In this way, during the specific execution process, the controller software does not need to intervene again, and the synchronization between task queues can be automatically completed on the hardware, greatly improving the parallel efficiency. In addition, the controller can also control and optimize the cooperation mode between the computing subtasks during task scheduling. For example, the result of the convolution performed by the TU computing unit must be activated by the CU. The startup of the CU computing unit depends on the completion of certain subtasks in the TU, but it is not necessary to wait for the completion of all network layer tasks in the TU. As long as the TU computing unit completes a part of the subtasks, the CU can execute tasks based on these partial results of the TU. This can be achieved by setting appropriate synchronization event points in each task queue.

[0045] In some embodiments, there may be three working modes between the various computing units: independent parallel mode, cooperative parallel mode, and interactive cooperative mode. Figure 2As shown in a), in the cooperative parallel mode, the computing subtasks of each computing unit are executed cooperatively in a pipeline manner, and the data dependency and synchronization of different computing subtasks are basically implemented in the on-chip memory shared by each computing unit. For example, TU and CU will cooperate to complete task 1 (such as batch processing 1), but TU can do it first, and then put its generated results in the on-chip memory (such as SPM), and then continue to execute task 2 (if the on-chip memory can retain more results, it can continue to execute subsequent tasks). The CU will extract the results generated by the TU from the on-chip memory, and then generate its own results for TU consumption. In this way, TU and CU will complete Task 1 and Task 2 together in a slightly decoupled manner. For example, the execution result of one layer of convolution of TU is immediately used by CU as the input of the next layer to perform the element-wise operation that it is good at, while TU can continue to do the same layer but work on another batch of data. As Figure 2 As shown in b), in the independent parallel mode, different computing units can basically execute independently and in parallel without urgent synchronization requirements. The data dependency and synchronization between the computing subtasks of each computing unit can be achieved through off-chip memory (such as DDR), because there is a large storage space to save related data. For example, TU and CU can each process a larger batch of data to fill nearly half of the DDR space, and then may switch to the data generated by each other for dependency to continue processing. Figure 2 As shown in Figure c), in the interactive cooperation mode, each computing unit needs to wait for each other during execution, for example, when the TU lacks certain capabilities but uses the CU to compensate, or when the CU lacks processing capabilities but uses the TU. In this mode, data dependency and synchronization are implemented on a cache memory with a limited size (such as an L2 cache) to minimize waiting time and improve TU / CU throughput. For example, the cache where the TU's results are temporarily stored is read by the CU to perform user-defined operations, and the results are put into the cache and further read by the TU to complete the fusion operation. These three modes can be mixed and used multiple times in the overall implementation of the neural network, and the working modes between the computing subtasks of each computing unit in the controller can be configured by software.

[0046] Continue to refer Figure 1, the memory access control module in the controller is mainly used to be responsible for the handling and control of the data involved in each computing subtask, and to control the input and output of the data operated by each computing unit. For example, the memory access control module can use a DMA component that is independent of the execution of each computing unit to pre-fetch the data involved in each computing subtask from the off-chip memory to the on-chip memory and / or cache memory through the memory access interface to minimize the number of memory accesses and the memory access waiting time. In some embodiments, the access control module can also be configured to determine the storage location and storage format of the intermediate results generated by each computing unit according to the operation mode between each computing unit, so that the data can be used directly and smoothly, avoiding or reducing additional data handling and reformatting. For example, in the independent parallel mode, the data generated by each computing unit is set to be stored in an off-chip memory (DDR) with a larger space. In the cooperative parallel mode, the data generated by each computing unit is set to be stored in the on-chip memory shared by each computing unit to minimize the number of memory accesses. In the interactive cooperative mode, the data generated by each computing unit is set to be stored in the internal cache to minimize the memory access waiting time. In some embodiments, the memory access control module may also configure and manage on-chip memory, cache memory, off-chip memory, or partial areas of the above memory that may be shared among computing units.

[0047] The following is an example of the controller task scheduling and synchronization process in conjunction with Figures 3(a) and (b), but it is not intended to be limiting. The controller receives a computational graph of a neural network to be processed, which can be predefined or converted by a software program, or obtained by commercial software such as TensorFlow. Each node in the computational graph of the neural network corresponds to a computational subtask to be executed, and the edges between the nodes represent the input and output relationship between the nodes, and also reflect the dependency relationship between the nodes. As shown in Figure 3(a), the controller assigns the computational subtasks corresponding to each node in the computational graph of the neural network to be processed to the corresponding task queues of different types of computing units (such as TU and CU) and processing units (such as DMA components in the memory access control module that are independent of each computing unit) according to the characteristics and capabilities of each current computing unit. For example, the computational subtasks DMA1, DMA2..., DMA5 related to accessing data are stored in the DMA task queue, and the tasks in the DMA task queue are executed by the DMA component in the memory access control module of the controller, and are used to access the data involved in the computational subtasks of each computing unit. The DMA component can be executed independently of each computing unit, and the handling and control of the load data flow can pre-fetch the data required for the computing subtask from the off-chip memory to the on-chip memory SPM to reduce the memory access waiting time. At the same time, the computing subtasks tu1, tu2..., tu5 such as matrix, vector, and convolution suitable for the TU computing unit are assigned to the task queue of the TU computing unit; and the computing subtasks not supported by the TU computing unit and the user-defined computing subtasks cu1, cu2..., cuN, cuN+1, cuN+2 are assigned to the task queue of the cu computing unit. As shown in Figure 3(b), when the controller assigns subtasks to each task queue according to the processing order, it also needs to consider the dependencies between the subtasks. When a subtask needs to rely on subtasks assigned to other task queues, a wait command can be inserted before the subtask, and the wait command can contain the task queue identifier and subtask identifier to be waited. When the wait command is executed, the task synchronization module checks whether the computing subtask indicated by the wait command has been executed. If the corresponding subtask has not been executed, it is necessary to continue waiting. In some other embodiments, when processing dependencies, the controller not only controls each task queue to execute its own tasks in parallel, but also allows the exchange of intermediate results between different task queues at certain synchronization points. For example, a task in a task queue can be set to be dependent on an event through hardware-level event synchronization technology, so that when a certain event notification is received, data generated by other types of computing units can be read from a cache area or a certain area of ​​on-chip memory.

[0048] In neural network processing, the neural network model is often abstracted into a directed acyclic data flow graph (i.e., a computation graph) composed of some operators, and then the kernel functions corresponding to these nodes are scheduled and executed in sequence according to a certain topological order, thereby completing the execution of the model. The operators corresponding to the nodes in the computation graph are usually defined at the granularity of algebraic operators, such as vector addition, subtraction, multiplication, division and matrix multiplication, etc. Due to the low abstract granularity of these operators, the computation graph of a neural network training model often includes thousands of nodes, and it is necessary to frequently call multiple kernel functions. The startup overhead of the kernel function and the data transmission overhead between kernel functions are relatively large. Therefore, in some other embodiments, the controller can also be configured to support the operator fusion mode, and the kernel functions corresponding to certain nodes in the computation graph are merged into one function, so that only one function call is required to complete it, thereby reducing the overhead caused by kernel startup. The controller can be configured to perform operation fusion on the same type of computing subtasks assigned to the same computing unit, so that the execution efficiency of the computing unit is more continuous and smooth, and the internal switching overhead is minimized and avoided. For example, if the TU computing unit supports both convolution and pooling operations, the convolution and pooling operations can be fused in the TU computing unit. If the TU computing unit does not support the pooling operation, the controller can instruct the small-scale convolution and pooling operations to be fused in the CU computing unit.

[0049] In some other embodiments, the controller can also be configured to support the inter-layer fusion mode of the neural network. At present, during the processing of the neural network model, each neural network layer reads the data output by the upper layer from the off-chip memory, and saves the processing results to the off-chip memory after processing the current layer as the input of the next layer. When the data batch is large, frequent access to the off-chip memory brings a lot of delays. In an embodiment of the present invention, the controller can be configured to support the inter-layer fusion mode, and the intermediate results of several consecutive network layer outputs can be saved in the on-chip memory SPM to reduce the number of off-chip memory accesses.

[0050] Reference Figure 4, which gives a structural schematic diagram of an on-chip heterogeneous artificial intelligence processor according to another embodiment of the present invention. The on-chip heterogeneous artificial intelligence processor includes multiple computing clusters connected by an on-chip data exchange network, wherein each computing cluster includes at least two computing units of different structural types, a memory access control module, and an on-chip memory shared by each computing unit, wherein the first type of computing unit is such as TU, and the second type of computing unit is such as CU, and the task queue corresponding to each computing unit is used to save the computing subtasks that need to be executed by the computing unit. The artificial intelligence processor also includes a controller. When the controller receives the computing graph corresponding to the neural network program to be processed, it performs functional analysis on the computing graph according to the current load conditions of each computing cluster and the characteristics of the different computing units contained in each computing cluster, decomposes it into multiple computing subtasks, and distributes each computing subtask to the corresponding task queue of the computing unit that can process the computing subtask of this type, and is used to control the synchronization between the computing subtasks. The artificial intelligence processor also includes a memory access interface for accessing off-chip memory, and a host interface for interacting with an off-chip main processor.

[0051] Each computing cluster and each computing unit in each computing cluster can have the three operating modes mentioned above: independent parallel mode, cooperative parallel mode and interactive cooperative mode. These three modes can be mixed and used multiple times in the overall implementation of the neural network, and the computing subtasks of each computing unit in the controller and the operating modes between each computing cluster can be configured by software. The computing clusters exchange data with each other through the on-chip data exchange network, and the global on-chip memory shared by each computing cluster can also be set according to actual needs.

[0052] In some embodiments, in the heterogeneous artificial processor of the aforementioned embodiment of the present invention, the controller can be configured through a configuration file, a configuration option or a software program. For example, the task scheduling module can decompose the computational graph of the neural network to be processed into multiple computational subtasks and distribute them to the task queues of the corresponding hardware of each computing unit according to the configuration option indicating the task allocation. These configuration options indicating task allocation may include task allocation based on type matching (selecting the computing unit that processes the subtask according to the computing subtask type), task allocation based on load (assigning tasks according to the queuing situation of the task queues of each computing unit), task allocation based on specified tags (distributing the computing subtask with a specified tag to the specific computing unit indicated by the tag), etc. One or more configuration options can be set at the same time, and different configuration options have different priorities, for example, the task allocation based on type matching has the highest priority, the task allocation based on the specified tag has the second highest priority, and the task allocation based on the load has the lowest priority. For another example, the task scheduling module can also set the working mode between each computing unit according to the configuration option indicating the working mode, such as independent parallel mode, cooperative parallel mode and interactive cooperative mode. In some other embodiments, the task scheduling module can also perform operation fusion on the computing subtasks assigned to the same computing unit according to the configuration option indicating operation fusion, and can notify the memory access control module to save the output of the intermediate neural network layer to the note-type memory according to the configuration option indicating inter-layer fusion. In some other embodiments, the computing graph nodes of the neural network to be processed can also be set or marked in advance by configuration files or software programs to determine which type of computing unit they are suitable for processing. In this way, when the task scheduling module receives the computing graph, it can distribute each computing subtask to the corresponding task queue of each computing unit according to these configuration options indicating task allocation. In addition, the dependency between each computing subtask can be configured, set or adjusted by configuration files or software programs. In this way, the task scheduling module can set the dependency between each computing subtask between each task queue according to the corresponding configuration options indicating the dependency between computing subtasks. In some embodiments, the task scheduling module can also be configured to set the working mode between each computing unit according to the received configuration options indicating the working mode. In some embodiments, the task scheduling module can also be configured to perform operation fusion on the computing subtasks assigned to the same computing unit according to the received configuration options indicating operation fusion. In some embodiments, the task scheduling module may also be configured to notify the memory access control module to save the output of the intermediate neural network layer to a scratchpad memory based on the received configuration option indicating inter-layer fusion.In the above embodiment, various configuration options can be used to flexibly adjust the distribution of computing subtasks between different computing units, the synchronization between task queues, and the data dependency and synchronization between computing subtasks, etc., so that the working mode of each computing unit can be flexibly set according to various application scenarios, so as to make fuller use of and give play to the hardware performance and computing efficiency of the heterogeneous processor itself.

[0053] References in this specification to "various embodiments," "some embodiments," "one embodiment," or "an embodiment," etc. refer to a particular feature, structure, or property described in conjunction with the embodiment being included in at least one embodiment. Thus, the appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., in various places throughout the specification do not necessarily refer to the same embodiment. Furthermore, particular features, structures, or properties may be combined in any suitable manner in one or more embodiments. Thus, particular features, structures, or properties shown or described in conjunction with one embodiment may be combined in whole or in part with features, structures, or properties of one or more other embodiments without restriction, as long as the combination is not illogical or inoperable.

[0054] The expressions of "including", "having" and similar meanings in this specification are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices. "One" or "an" does not exclude multiple situations. In addition, the elements in the drawings of this application are for illustrative purposes only and are not drawn to scale.

[0055] Although the present invention has been described through the above embodiments, the present invention is not limited to the embodiments described herein, but also includes various changes and modifications made without departing from the scope of the present invention.

Claims

1. A configurable heterogeneous artificial intelligence processor, comprising at least two computing units of different structural types and a task queue, a storage unit and a controller corresponding to each computing unit; wherein the computing units of different structural types refer to computing units using different types of instruction sets and architectures, one of the at least two computing units of different structural types is a customized computing unit for a specific artificial intelligence algorithm or operation, and the other is a programmable computing unit; the controller comprises a task scheduling module, a task synchronization module and a memory access control module; in, The controller is controlled by a configuration file, wherein the configuration file includes a first plurality of configuration options indicating task allocation, a second plurality of configuration options indicating a working mode, and a third plurality of configuration options indicating dependencies between computing subtasks; The task scheduling module is used to decompose the computation graph of the neural network to be processed into multiple computation subtasks and distribute them to the corresponding task queues of each computing unit according to the first plurality of configuration options indicating task allocation in the configuration file, in, The first plurality of configuration options include: A first configuration option is used to indicate the selection of a computing unit for processing a computing subtask according to the type of the computing subtask; The second configuration option is used to instruct to allocate tasks according to the queuing status of the task queues of each computing unit; A third configuration option is used to instruct to distribute the computing subtask to the computing unit indicated by the tag according to the specified tag of the computing subtask; and wherein the first configuration option, the second configuration option, and the third configuration option have respective priorities; The task scheduling module is further used to set the dependency relationship between the computing subtasks according to a third plurality of configuration options indicating the dependency relationship between the computing subtasks in the configuration file; and Setting the working mode between the computing units according to the second plurality of configuration options indicating the working mode in the configuration file, wherein the working mode includes an independent parallel mode, a cooperative parallel mode and an interactive cooperative mode; in, In independent parallel mode, the computational subtasks of each computing unit are executed independently and in parallel; In cooperative parallel mode, the computational subtasks of each computing unit are executed cooperatively in a pipeline manner; In the interactive cooperation mode, when executing a computing subtask, a computing unit needs to wait for or rely on the results generated by the computing subtasks executed by the computing units that cooperate with it; The task synchronization module realizes synchronization between various computing subtasks according to the set dependency relationship; The memory access control module is used to control the access of data involved in each computing subtask to the storage unit and the off-chip memory, and the storage unit includes a cache memory and a scratchpad memory; The memory access control module is further configured to set the storage location of the shared data between different computing units according to the working mode between the computing units. in, In independent parallel mode, this storage location is set in off-chip memory; In cooperative parallel mode, the storage location is set on the scratchpad memory; In interactive cooperation mode, the storage location is set on the cache memory.

2. The heterogeneous artificial intelligence processor according to claim 1, wherein the task scheduling module is further configured to perform operation fusion on computing subtasks assigned to the same computing unit according to a configuration option indicating operation fusion.

3. The heterogeneous artificial intelligence processor according to claim 1, wherein the task scheduling module is further configured to notify the memory access control module to save the output of the intermediate neural network layer to the scratchpad memory according to the configuration option indicating inter-layer fusion.

4. The heterogeneous artificial intelligence processor according to claim 1, wherein the structure type of the computing unit is one of the following: application-specific integrated circuit ASIC, general-purpose graphics processing unit GPGPU, field programmable gate array FPGA, video codec DSP.

5. The heterogeneous artificial intelligence processor according to claim 1, wherein the at least two computing units of different structural types include computing units using an application-specific integrated circuit ASIC structure and computing units using a general-purpose graphics processing unit GPGPU structure.

6. The heterogeneous artificial intelligence processor according to claim 1, wherein the task scheduling module is further configured to distribute computing subtasks to corresponding computing units according to the capabilities of each computing unit.

Images