Reconfigurable neural engine with extensible instruction set architecture

Abstract

The integrated circuit includes a plurality of kernels and a virtual machine coupled to the plurality of kernels, the virtual machine configured to interpret instructions directed to different kernels of the plurality of kernels, and the virtual machine configured to control operation of the different kernels of the plurality of kernels in response to the instructions.

Description

[Technical Field]

[0001] (Reservation of Rights for Copyrighted Works) A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction of either this patent document or the patent disclosure, as appearing in the Patent and Trademark Office patent file or patent records, but otherwise reserves all copyright rights whatsoever.

[0002] FIELD OF THE INVENTION The present disclosure relates to artificial neural networks, and more particularly to a reconfigurable neural engine with an extensible instruction set architecture for implementing artificial neural networks in integrated circuits. [Background technology]

[0003] Deep learning refers to a subset of machine learning. To accomplish a given task, deep learning utilizes artificial neural networks. An artificial neural network, or simply a "neural network," includes layers of interconnected nodes operable to classify input data into categories of interest. Generally, a neural network includes an input layer and an output layer, and optionally, one or more hidden layers between the input and output layers. Neural networks represented as machine learning models used in deep learning applications typically include many hidden layers, giving rise to the term "deep neural network."

[0004] Machine learning models can be developed that specify particular neural networks. Inference refers to calculating the output of a machine learning model given input data. Because inference is computationally intensive, significant development efforts have been made to run machine learning models more efficiently on various hardware platforms (e.g., central processing units, graphics processing units, field programmable gate arrays, digital signal processors, and application specific integrated circuits).

[0005] In some cases, machine learning models are implemented by spatially distributing fixed-function kernels over available hardware resources of a hardware platform. Generally, a kernel refers to a design unit that performs an operation in a machine learning model. For example, a kernel can perform functions such as common matrix multiplication, aggregation, pooling, or activation. Spatially distributing kernels over a hardware platform can consume a significant amount of time and, upon completion, can still result in the placement and routing of fixed-function kernels over the hardware platform, which creates a performance bottleneck.

[0006] In other cases, developers create hand-tuned kernels that are spatially distributed over the available hardware resources of a hardware platform. Creating hand-tuned kernels is also a time-consuming and error-prone endeavor. Hand-tuned kernels may not be compatible with other machine learning models and are often coded at a low level to efficiently utilize the hardware resources of the underlying hardware platform. This can make it difficult to scale up hand-tuned kernels. Summary of the Invention [Means for solving the problem]

[0007] In one or more exemplary implementations, an integrated circuit includes a plurality of kernels and a virtual machine coupled to the plurality of kernels. The virtual machine is configured to interpret instructions directed to different kernels of the plurality of kernels. The virtual machine is configured to control the operation of the different kernels of the plurality of kernels in response to the instructions.

[0008] In one or more exemplary embodiments, a method includes receiving a first instruction by a virtual machine running on electronic hardware. The method includes using the virtual machine to analyze the first instruction to determine a first kernel from multiple kernels coupled to the virtual machine. The method includes the virtual machine configuring the first kernel using configuration data to perform an operation specified by the first instruction. The configuration data specifies a buffer containing input data to the first kernel and a buffer for storing data generated by the first kernel. The method includes using the virtual machine to cause the first kernel to perform the operation as configured.

[0009] This Summary of the Invention section is provided merely to introduce certain concepts and is not intended to identify any key or essential features of the claimed subject matter. Other features of the inventive structure will be apparent from the accompanying drawings and the detailed description that follows.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS Configurations of the present invention are illustrated by way of example in the accompanying drawings. However, the drawings should not be construed as limiting the configuration of the present invention to only the particular embodiments shown. Various aspects and advantages will become apparent upon consideration of the following detailed description and upon reference to the drawings. [Brief explanation of the drawings]

[0011] [Figure 1] Illustrates an example of a reconfigurable neural engine. [Figure 2] 1 illustrates an example of a static instruction format that may be used by a reconfigurable neural engine virtual machine. [Figure 3] 1 illustrates an example of a dynamic instruction format that may be used by a reconfigurable neural engine virtual machine. [Figure 4] 1 illustrates an exemplary memory layout for use with an instance of a reconfigurable neural engine. [Figure 5]1 illustrates another example of a memory layout for an instance of a reconfigurable neural engine where memory management occurs during runtime. [Figure 6] 1 illustrates another example of a memory layout for an instance of a reconfigurable neural engine where memory management occurs during runtime. [Figure 7] 1 illustrates an example of an integrated circuit capable of implementing one or more instances of a reconfigurable neural engine. [Figure 8] 1 illustrates an example of a computational tile of a data processing array in which an instance of a reconfigurable neural engine can be implemented. [Figure 9] 1 illustrates an exemplary method of operation of a reconfigurable neural engine. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present disclosure relates to artificial neural networks, and more particularly to a reconfigurable neural engine with an extensible instruction set architecture for implementing artificial neural networks in integrated circuits. In accordance with the inventive features described within this disclosure, the reconfigurable neural engine provides an architectural standard for integrating and / or using kernels. The reconfigurable neural engine can be used by higher-level compilers and / or machine learning frameworks to design, control, and execute kernels capable of performing machine learning operations for machine learning models on a selected hardware platform.

[0013] In one or more examples, a reconfigurable neural engine can interact with multiple kernels. The kernels may be functionally independent and implement different machine learning functions. A machine learning function is a function performed by a type of layer of a machine learning model implemented as a multilayer artificial neural network. Examples of different types of machine learning functions that may be implemented as kernels include convolution, general matrix-multiply (GEMM), BiasAdd (adding a bias to a value), aggregation, pooling, normalization, requantization, and activation functions such as rectified linear unit (ReLu), softmax, hyperbolic tangent (tan h), and identity function.

[0014] Using a reconfigurable neural engine, multiple kernels can be assembled to execute within the given hardware resources of a hardware platform. In one aspect, the reconfigurable neural engine acts as glue logic that provides a standardized framework to control kernels coupled to a virtual machine implemented as part of the reconfigurable neural engine. The virtual machine provides a standardized interface for any kernels coupled to it. The standardized interface provides flexibility in that kernels can be added and removed (e.g., swapped) without disrupting the reconfigurable neural engine's overall software stack.

[0015] The reconfigurable neural engine allows kernels to execute independently of one another. The reconfigurable neural engine facilitates efficient use of the underlying hardware resources of the hardware platform while also providing software-level programmability of the kernels. The reconfigurable neural engine can provide improved computational power and improved memory utilization compared to other kernel implementation techniques. The reconfigurable neural engine implements a control mechanism that allows any high-level framework to program and control the kernels individually. Further aspects of the inventive arrangement are described below with reference to the figures.

[0016] 1 illustrates an example of a reconfigurable neural engine 100. Reconfigurable neural engine 100 refers to a particular implementation or instance of a virtual machine 102 coupled to one or more kernels 104 (e.g., 104-1, 104-2, ..., 104-N). Virtual machine 102 can receive input data 106, input data 108, and one or more instructions via instruction stream 110. In an exemplary implementation, each of input data 106, 108 may represent a window of data of a particular size. For purposes of illustration and not limitation, input data 106 and input data 108 may each be a 2 x 8 kB window of data (e.g., the windows are double buffered in this example).

[0017] The reconfigurable neural engine 100 can perform memory management for the kernels 104 by coordinating access of different ones of the kernels 104 to different ones of the buffers 112 (e.g., 112-1, 112-2, ..., 112-N). The buffers 112 can be local memory buffers for the kernels 104. The kernels 104 can operate on input data 106 and / or 108 stored in the buffers 112 and generate results that are output as output data 114. The output data 114 can be an output window or a data stream.

[0018] In the example of FIG. 1 , kernels 104 may implement any of a variety of different machine learning functions. For example, kernel 104-1 may be a GEMM kernel. Kernel 104-2 may be a BiasADD kernel. Another kernel 104 may be a ReLU kernel. Yet another kernel 104 may be a requantization kernel (e.g., a kernel capable of performing shift and scale functions). Still other kernels 104 may implement other machine learning functions, such as a Gaussian Error Linear Unit (GELU), layer normalization, Softmax, etc. The example machine learning functions provided herein are intended to be a non-exhaustive list of machine learning functions that may be implemented by kernels 104.

[0019] Virtual machine 102 may receive instructions (e.g., as instruction stream 110) from another circuit and / or system. Virtual machine 102 may interpret the received instructions. In response to the instructions and their interpretation, virtual machine 102 may control the operation of kernels 104. Virtual machine 102's interpretation of instructions makes kernels 104 programmable. More specifically, in response to the received instructions, virtual machine 102 may select one or more of kernels 104, configure certain aspects of the selected kernels 104, manage local memory buffers of the selected kernels 104, execute the selected kernels 104, and potentially return data generated by the selected kernels 104 (e.g., as output data 114).

[0020] In one or more examples, virtual machine 102 is implemented as a state machine configured to receive and act on instruction stream 110 generated and / or provided by another circuit and / or system. Virtual machine 102 can parse incoming instructions to determine the size of the instruction packet, determine which kernel to execute based on the opcode parsed from the instruction, determine memory availability and reuse, and allocate buffers 112 to the kernel to be executed as specified by the opcode. In this regard, reconfigurable neural engine 100 is compatible with any higher-level framework (e.g., circuit and / or system) that can support the instruction format used by reconfigurable neural engine 100 and, in particular, virtual machine 102.

[0021] The use of a reconfigurable neural engine 100 provides a higher level of abstraction compared to traditional techniques for executing kernels. This abstraction allows kernels 104 used with the reconfigurable neural engine 100 to be scaled more easily, solving the placement and routing problems associated with using fixed-function kernels. Kernel reuse is also supported in that the reconfigurable neural engine 100 provides a standardized interface for kernels 104, which facilitates a plug-and-play development paradigm. That is, kernel 104-1 can be replaced with another kernel 104 that performs different functions without significant reimplementation of the software stack, as long as the new or different kernel is able to interface with the virtual machine 102 for the kernel (e.g., by adhering to the standard interface provided by that virtual machine).

[0022] The reconfigurable neural engine 100 also provides a high degree of portability between different kernels. For example, another circuit and / or system can be updated to generate compatible instructions for the virtual machine 102. The virtual machine 102 can interpret the received instructions to control, adjust (e.g., configure), and execute any kernels coupled to it. However, the lower-level implementation details of the kernels may remain unchanged.

[0023] The example of FIG. 1 can be implemented using any of a variety of different hardware platforms. In one aspect, the hardware platform is implemented as a data processing array. An example of a data processing array is implemented as multiple hardwired circuit blocks. The multiple circuit blocks may be programmable. The data processing array may include multiple computational tiles and interface blocks. Generally, a computational tile includes a core having data processing capabilities and local (L1 cache) memory that may be used to implement buffer 112. The local memory may also be accessed, subject to constraints, by other cores in the data processing array. The data processing array may include shared memory (L2 cache, also referred to as memory tile) accessible by all cores in the data processing array. An exemplary data processing array is described in connection with FIG. 7. A data processing array such as that illustrated in FIG. 7 can be programmed to perform one or more operations of a deep neural network.

[0024] In one aspect, the reconfigurable neural engine 100 can be implemented within a computational tile of a data processing array. For example, the reconfigurable neural engine 100 can be deployed across a core of a computational tile of the data processing array. In another aspect, multiple instances of the reconfigurable neural engine 100 can be deployed across multiple cores (e.g., across multiple computational tiles) of the data processing array. When the reconfigurable neural engine 100 is running on multiple computational tiles, another circuit and / or system (e.g., capable of running a high-level framework such as a controller and / or processor) can be used to schedule instructions and data for the multiple computational tile implementation.

[0025] In another aspect, the hardware platform may be implemented as a programmable circuit. The reconfigurable neural engine 100 may be implemented within a programmable circuit. The programmable circuit may include or be implemented as programmable logic. An example of a programmable circuit is described in connection with FIG. 7. In yet another example, the reconfigurable neural engine 100 may be implemented as a combination of a processor (e.g., a hard or soft processor) and a programmable circuit that implements the kernels.

[0026] In the example of FIG. 7 , a single IC may provide multiple different types of hardware platforms on which reconfigurable neural engine 100 may be implemented. It should be understood that the inventive arrangements may be used with other types of ICs, including data processing arrays alone, data processing arrays in combination with one or more other systems, programmable circuits alone, and / or programmable circuits in combination with one or more other systems. Furthermore, a machine learning design may be realized using one or more instances of reconfigurable neural engine 100 implemented in one type of hardware platform, with one or more other instances of reconfigurable neural engine 100 implemented in a different type of hardware platform. In yet other examples, reconfigurable neural engine 100 may be implemented using a single processor or multiple processors. Furthermore, different implementations may operate cooperatively as part of the same or common machine learning design implemented within an IC.

[0027] In one or more examples, two (or more) different implementations of the reconfigurable neural engine 100 may be implemented. For illustrative purposes, in an example, a computational tile may execute two (or more) different implementations of the reconfigurable neural engine 100. A first implementation of the reconfigurable neural engine 100 may include kernels A, B, and C configured to operate on mixed-precision data. A second implementation of the reconfigurable neural engine 100 may include kernels configured to operate on integer data (e.g., int8). For example, kernels in the second implementation of the reconfigurable neural engine 100 may perform matrix multiplication, BiasADD, ReLU, etc. The second implementation of the reconfigurable neural engine 100 may provide improved latency and throughput compared to the first implementation of the reconfigurable neural engine 100 capable of operating on mixed-precision data.

[0028] The foregoing example using multiple different reconfigurable neural engine 100 implementations illustrates one way in which the reconfigurable neural engine 100 is able to support evolving computational algorithms and data flow throughput.

[0029] As described above, each kernel in reconfigurable neural engine 100 may execute independently under the control of virtual machine 102 in response to instructions provided to virtual machine 102. The instructions may be generated by an entity such as another circuit and / or system (e.g., a controller and / or processor) that implements a high-level framework such as Tensor Virtual Machine (TVM), PyTorch, TensorFlow, etc. These instructions may be passed from that entity to various reconfigurable neural engine 100 implementations and / or instances during execution.

[0030] The instructions received and interpreted by the virtual machine 102 may include configuration data that can be used by the virtual machine 102 to configure each kernel 104. That is, the virtual machine 102 may receive the instructions. From the instructions, the virtual machine 102 may determine a particular kernel 104 appropriate for the instruction, configure the kernel according to predefined rules within the virtual machine 102 and / or configuration data specified by the instruction, and invoke or execute the kernel. The configuration data, whether hard-coded within the virtual machine 102 or obtained from the received instructions, may be provided to the kernel specified by the instruction to cause the kernel to operate in a particular operating mode, which may be one of multiple different operating modes for the selected kernel 104.

[0031] As an example, consider the case where kernel 104-1 implements a linear approximation kernel. That is, kernel 104-1 may implement any one of several different functions (e.g., activation functions) through linear approximation. In that case, virtual machine 102 determines the configuration data specified by the received instruction specifying the linear approximation kernel, provides the configuration data to the kernel, and invokes the kernel. In this example, the configuration data may specify different coefficients used by the kernel in performing the operation. The particular coefficients may be specific to the operation mode and the linear approximation being performed. Thus, a particular operation mode of a kernel can be invoked by passing different instructions to virtual machine 102 that have different configuration data but are targeted to the same kernel.

[0032] In one aspect, the reconfigurable neural engine 100 can be instantiated with a machine learning application specified as a graph. More specifically, the reconfigurable neural engine 100 can be instantiated with an application graph class in a separate header file, similar to how a kernel is instantiated. The header file can connect the reconfigurable neural engine 100 (e.g., the virtual machine 102) to other kernels and provide dataflow information using an available application programming interface (API) in a development tool. The standardized interface for the kernels provided by the virtual machine 102 allows the kernels to be reused with different machine learning models. Furthermore, the kernels can be reconfigured through high-level instructions received by the virtual machine 102.

[0033] The example source code in Listing 1 illustrates an example of instantiating a reconfigurable neural engine 100 through a graph application programming interface (API). In the example in Listing 1, a GMIO port is used to pass data between device memory (e.g., Double Data Rate Synchronous Dynamic Random Access Memory (DDR)) and the reconfigurable neural engine 100. Device memory refers to memory that is global memory accessible to an instance of the reconfigurable neural engine 100 as implemented in the hardware platform. GMIO is a class that represents an I / O port attribute specification used to make an external memory-mapped connection to or from global memory (e.g., device memory). In the example source code in Listing 1, "arne" represents the reconfigurable neural engine 100.

[0034] [Table 1]

[0035] As illustrated in Listing 1, reconfigurable neural engine 100 can support any kernel data access type, including window-based data access through input_window and output_window as illustrated in Listing 1. Additionally, reconfigurable neural engine 100 can support stream-based access, such as input_stream and output_stream in Listing 1. The types of data access supported by reconfigurable neural engine 100 (e.g., virtual machine 102) can include any data type access supported by the kernel.

[0036] The example source code in Listing 2 illustrates an implementation of a reconfigurable neural engine 100 that includes one stream instruction, two input windows for providing input data 106 and 108, and one output window for providing output data 114.

[0037] [Table 2]

[0038] In another aspect, the reconfigurable neural engine 100 can use multiple stream connections shared on a single physical channel. Multiple stream connections can be split using a packet switching structure. The example source code in Listing 3 illustrates the use of a "pktsplit" structure to split multiple input stream connections sharing a single physical channel into instruction and data windows. The pktsplit structure in the example of Listing 3 splits a single buffer into an instruction buffer and a data buffer that go to different inputs of the virtual machine 102.

[0039] [Table 3]

[0040] The virtual machine 102 may be implemented to support a particular instruction format. In an exemplary implementation, the virtual machine 102 is implemented to use a static instruction format.

[0041] 2 illustrates an example of a static instruction format 200 that may be used by virtual machine 102. The instruction format is called fixed because the instructions have a fixed length. In the example of FIG. 2, instruction format 200 is implemented as a vector including four fields, including an opcode field 202, a settings field 204, a data field 206 (e.g., data0), and a data field 208 (e.g., data1). In one aspect, each field may be specified as an integer data type of a particular size, such as 8, 16, or 32 bits. It should be understood that the example sizes are provided for purposes of illustration and are not intended to be limiting.

[0042] The opcode field 202 specifies a particular kernel to be executed. For example, the virtual machine 102 may include a pre-configured opcode-to-kernel map. For a given instruction, upon parsing the opcode from the instruction, the virtual machine 102 determines the particular kernel 104 to be executed in response to the instruction. For example, an opcode of "0" may be mapped to a general matrix multiplication kernel. An opcode of "1" may be mapped to a ReLU kernel. An opcode of "2" may be mapped to a BiasAdd kernel, and so on.

[0043] The configuration field 204 may be a customizable field, i.e., a kernel designer may define the particular operational modes in which their kernel may operate. The configuration field 204 may contain configuration data that specifies the particular operational mode to be implemented by the kernel specified by the opcode field 202.

[0044] As an example, the least significant bit (LSB) of the configuration field 204 may be used to indicate whether the kernel will output a value for an iteration (e.g., execution) of the kernel initiated by the instruction. As another example, the second least significant bit of the configuration field 204 may be used to indicate whether the output of the kernel should be shifted. Further data bits of the configuration field 204 may specify by how many bits to shift the data if a shift should be applied.

[0045] The configuration field 204 may be customized for each kernel. That is, the configuration data contained in the configuration field 204 may be kernel-specific. Thus, the meaning of the bits in the configuration field 204 of an instruction for kernel 104-1 may be different from the meaning of the bits in the configuration field 204 of an instruction for kernel 104-2. It can be appreciated that the virtual machine 102 can correctly interpret the configuration bits / data specified in the configuration field 204 based on the particular opcode specified in the received instruction.

[0046] Data fields 206, 208 specify addresses of data, e.g., buffers, that may be used by the kernel. In one aspect, one or more bits of data in configuration field 204 may specify what each of data fields 206, 208 is to be used for. That is, based on the data bits of configuration field 204, data field 206 may specify an input buffer (e.g., input data 106), data field 208 may specify a different input buffer (e.g., input data 108), and data field 208 may specify an accumulation buffer, scratch space, and / or an output buffer (e.g., output data 114). Thus, the data bits of configuration field 204 allow a kernel designer to specify the purpose of the data specified by one or both of data fields 206, 208. In this example, using static instruction format 200, memory management functions may be implemented by using data fields 206 and / or 208 to specify various types of buffers to be used by the kernel.

[0047] The example source code in Listing 4 may be incorporated into the virtual machine 102 and illustrates an example where the virtual machine 102 determines whether the output of a GEMM kernel should be shifted based on data bits in the configuration field 204. The example in Listing 4 determines whether the output of a GEMM kernel should be shifted right and by how many bits.

[0048] [Table 4]

[0049] The static instruction format 200 of FIG. 2 provides several advantages, such as reduced implementation complexity of the virtual machine 102.

[0050] In one or more other examples, the reconfigurable neural engine 100 may be implemented with a virtual machine 102 configured to parse a dynamic instruction format. The dynamic instruction format may be specified as a base class for instructions that includes fields shared by all kernels in the reconfigurable neural engine 100. Derived classes may specify kernel-specific configuration and data fields.

[0051] The example source code in Listing 5 illustrates an example of a dynamic instruction header file for a machine learning model including GEMM, requantization, loopback, and general nonlinear kernels. The example source code in Listing 5 illustrates a base class "op_base." The base class "op_base" includes shared fields such as size_in_bytes, which specifies the size of the instruction in bytes, is_init, which is used for kernel initialization, and is_wait, which is used for kernel control. The base class is instantiated in kernel-specific instructions and derived classes along with custom configuration and data fields. For example, custom data fields is_shift and shift_val are specified for a GEMM instruction directed to a GEMM kernel. By providing size_in_bytes through the base class, the virtual machine 102 can determine how many bytes are used for each kernel's instruction during execution. The shared configuration field can be decoded by the virtual machine 102 before passing the instruction's custom configuration and data fields to the specified kernel.

[0052] [Table 5]

[0053] In another aspect, a dynamic instruction format may be used to manage local memory for the kernel. The dynamic instruction format may specify one or more bits that define the input buffer to be used for a given instruction and / or the output buffer to be used for a given instruction.

[0054] FIG. 3 illustrates an example of a dynamic instruction format 300. In the example of FIG. 3, the dynamic instruction format 300 includes a size field 302, an opcode field 304, one or more shared settings fields 306, one or more custom settings fields 308, and one or more custom data fields 310. The size field 302 may be a size_in_bytes field. The opcode 304 may specify a particular kernel to be executed, as described in connection with FIG. 2. By way of illustration and not limitation, in the example of FIG. 3, the opcode field 304 may specify a kernel such as GEMM, ReLu, BiasAdd, or requantize. The shared settings field 306 may specify bits indicating the status of is_init, is_out, and / or bits specifying particular input and / or output buffers. The custom settings field 308 may specify additional configuration fields for each opcode. For example, for an opcode specifying a requantize kernel, the custom settings field 308 may specify configuration data such as a multiplier, a Q number, and / or an amount to shift data.

[0055] In the example of FIG. 3, a kernel may execute multiple times before outputting data. If a kernel is executed four times, for example, four instructions for the kernel are received by the virtual machine 102. The shared settings field 306 may contain different data for different iterations or executions of the kernel. The is_init bit may be used to indicate whether a specified buffer should be zeroed (e.g., initialized). The is_wait bit may be used in the instruction to indicate whether the kernel is outputting data (e.g., in the last iteration) or whether it is storing data in a different buffer for reuse in the next iteration or execution (e.g., the first three of four iterations). Other bits may be used to indicate whether and by how much the kernel should shift data for a given execution.

[0056] FIG. 4 illustrates an exemplary memory layout 400 for use with an instance of a reconfigurable neural engine 100. The memory layout 400 may be implemented within a computational tile of a data processing array. For example, the kernel 104 may be loaded into the program memory of the computational tile. More specifically, the kernel 104 may be loaded into the program memory of a core of the computational tile. The remainder of the memory layout 400 may be implemented within the RAM (e.g., level 1 cache) of the computational tile so as to be distributed across banks 1, 2, 3, and 4. Examples of computational tiles are described in conjunction with FIGS. 7 and 8.

[0057] For purposes of illustration, in the example reconfigurable neural engine 100, kernel 104-1 is a GEMM kernel, kernel 104-2 is a requantization kernel, and kernel 104-3 is a layer normalization kernel. Kernels 104 are loaded into and executed from computational tile program memory. This instance of reconfigurable neural engine 100 is provided for purposes of illustration and not limitation. Other implementations of reconfigurable neural engine 100 may include fewer or more kernels than those illustrated, and / or different kernels.

[0058] In this example, the shaded portions of memory layout 400 correspond to active or used regions of memory. In this example, kernel 104-1 is executing and using the shaded regions of memory in the computation tile RAM. Region 402 stores buffer A (e.g., input matrix for kernel 104-1), and region 406 stores buffer B (e.g., input matrix B for kernel 104-1). Regions 404 and 408 store data for the next execution of kernel 104-1, illustrated as buffers Ad and Bd, respectively. In this example, banks 1 and 2 are each divided into two regions so that data can be loaded into buffers A and B in a ping-pong fashion, and then into buffers Ad and Bd. Thus, while kernel 104-1 is operating on buffers A and B, buffers Ad and Bd can be loaded into regions 404 and 408, respectively. For example, buffer A (and buffer Ad) can correspond to input data 106, and buffer B (and buffer Bd) can correspond to input data 108.

[0059] Region 410 stores the accumulation (ACC) buffer for kernel 104-1. Region 412 stores another accumulation buffer (ACCd) for kernel 104-1 that can be used in a ping-pong fashion with the ACC buffer of region 410. Region 418 stores buffer C for kernel 104-1, e.g., an output buffer corresponding to output data 114. Region 414 stores temporary buffers that can be used for kernel 104-1 during execution. The temporary buffers may be overwritten by another kernel, such as kernel 104-3, when that kernel executes. Region 416 can be used to store buffers containing configuration parameters for layer normalization kernels (e.g., kernel 104-3). Region 420 can store stack and heap data. Region 422 can be used to store intermediate results as "scratch" data.

[0060] When using the reconfigurable neural engine 100, kernels may be used in a "plug-and-play" manner. That is, one kernel may be removed and replaced (e.g., "swapped out") with another, different kernel. The two kernels may or may not have the same memory footprint. Thus, in one aspect, to increase portability and reduce engineering time, memory management for the reconfigurable neural engine 100 may be determined at creation time.

[0061] The example of Figure 4 illustrates how a static memory layout can be used when all of the data required by each kernel fits into available memory (e.g., in computational tile RAM). While it may be possible to organize all kernels to use available memory for their operations, this static memory layout may reduce the reusability of the reconfigurable neural engine 100 for input data of different sizes to different kernels, reduce kernel portability, and reduce the kernel's plug-and-play capabilities. As illustrated in the example of Figure 4, tightly packed data utilizing all of the available computational tile RAM for three kernels 104 would have to undergo a significant redesign if one of the kernels needed more memory to store configuration parameters or if additional kernels were added. However, if sufficient space is available in memory, one or more additional kernels can be added to a given reconfigurable neural engine 100 implementation without perturbing the memory layout.

[0062] Figure 5 illustrates another example of a memory layout 500 in which memory management occurs during execution. In the example of Figure 5, memory management may be performed by the virtual machine 102 in response to instructions received by the virtual machine 102.

[0063] In one example, the shared settings field 306 can be used to associate a region of memory with a memory map maintained by the virtual machine 102. The memory map tracks the state of different regions of memory and indicates whether a given region of memory can be reused or reclaimed, for example, by different kernels. For example, a (1) for a given region of memory in the memory map indicates that the region can be reused. A (0) for a given region of memory in the memory map indicates that the region cannot be reused. The memory map can also specify a pointer to the start of a buffer stored in a given region and the size of the buffer stored in that region.

[0064] Referring to FIG. 5, the shaded region of memory is currently in use by kernel 104-1, which is also shaded. In this example, kernel 104-1 is running. For illustrative purposes, region 416 does not store layer normalization parameters but stores other data. Region 414 stores requantization buffers for use by kernel 104-2. In the example of FIG. 5, region 402 initially stored buffer A for use by kernel 104-1. Because the memory map maintained by virtual machine 102 indicated that buffer A was available for reuse (e.g., the memory map includes an entry such as "mem_ready[GEMM_BUFF_A]==1," indicating that buffer A for kernel 104-1 is available for reuse), virtual machine 102 overwrites buffer A in region 402 with layer normalization parameters that may be used by kernel 104-3 when kernel 104-3 executes.

[0065] In the example of FIG. 5 , virtual machine 102, upon determining that kernel 104-1 no longer needs buffer A because the memory map indicates a status of “1” for buffer A, loads different data in region 402, effectively reclaiming region 402 for use by another kernel. In this example, a higher-level framework providing instructions to virtual machine 102 can determine when the GEMM operation performed by kernel 104-1 is complete or that data from region 402 has been read by kernel 104-1 and is no longer needed in memory. The ability to reclaim regions of memory improves memory reuse to support kernels that may need to store various types of data, such as lookup tables, or store large amounts of intermediate results for processing. Reclaiming allows use of a particular region of memory to be reallocated from a first kernel to a second kernel before the first kernel completes execution, thereby improving runtime efficiency.

[0066] In addition to facilitating memory reuse, the memory management functionality of virtual machine 102 also allows kernels to share data with each other within the same computational tile. For example, kernels executed by the same core (e.g., within the same computational tile) can share input and / or output data with each other. Sharing data between kernels may be implemented by providing instructions to the kernels that cause them to read specific memory buffers for input and / or store output in specific (e.g., different) memory buffers.

[0067] The example code in Listing 6 illustrates portions of two example instructions with dynamic instruction formats.

[0068] [Table 6]

[0069] The first instruction includes an opcode (opcode, OP) that selects the GEMM kernel. The first instruction includes an input portion (input, IN) that specifies a buffer from which the GEMM kernel reads input data. In this example, the IN portion specifies a value of "1" for input buffers A and B. The output portion (output, OUT) of the first instruction includes a "1" that corresponds to the accumulation buffer, which causes the GEMM kernel to write output to the accumulation buffer. In response to instruction 1, virtual machine 102 configures the GEMM kernel to perform the operation using the specified buffer and executes the kernel as configured.

[0070] The second instruction includes an opcode (OP) that, in this example, selects a requantization kernel. The second instruction includes an input portion (IN) that specifies a buffer from which the requantization kernel will read input data. In this example, the IN portion specifies a value of "1" for the accumulation buffer. The output portion (OUT) of the second instruction includes a "1" that corresponds to buffer C, which causes the requantization kernel to write output to buffer C. In response to instruction 2, virtual machine 102 configures the requantization kernel to operate using the specified buffer as configured.

[0071] The example in Listing 6 illustrates how the virtual machine 102, when processing a received instruction, continues to have a first kernel (e.g., a GEMM kernel) write data to a selected buffer (e.g., an accumulator buffer) instead of output buffer C. In this example, the GEMM kernel is not outputting data, but rather stores the data in a different buffer that another kernel can use. Next, or subsequently, a second kernel (e.g., a requantization kernel) operates on the selected buffer (e.g., an accumulator buffer) as input data and stores the resulting data in output buffer C (e.g., output data 114), which can be communicated to another circuit and / or system via a stream channel. This feature provides greater control over memory layout at runtime and more efficient use of the memory layout. Furthermore, the need to redesign machine learning models by determining new and / or different memory layouts when trying different kernel options is reduced.

[0072] 6 illustrates another example of a memory layout 500 in which memory management occurs during execution. In the example of FIG. 6, the memory map indicates that buffer B is reusable. That is, because the memory map maintained by virtual machine 102 indicated that buffer B is reusable (e.g., the memory map includes an entry such as "mem_ready[GEMM_BUFF_B]==1," indicating that buffer B for kernel 104-1 is reusable), virtual machine 102 has overwritten buffer B in region 402 with layer normalization parameters that may be used by kernel 104-3 when kernel 104-3 executes.

[0073] In order for a kernel to be integrated into a particular reconfigurable neural engine 100 implementation, the kernel must conform to the interface requirements of the virtual machine 102. The virtual machine 102 is only responsible for interacting with input and output data on behalf of any integrated kernels. That is, input data can only be provided to the kernel via input data 106, 108, each of which may be an input window of a particular size, and through a stream interface through which the input data 106, 108 is conveyed. Kernels integrated into a reconfigurable neural engine 100 implementation must use pointers to access the contents of local memory (e.g., to compute the tile RAMs of Figures 4, 5, and / or 6).

[0074] The example source code in Listing 7 illustrates a scenario in which kernels use pointers to access local memory. In the example of Listing 7, each kernel (e.g., ReLU, BiasAdd, and GEMM) uses T to access the input and output buffers. * It is defined using the following: * " is the indirection operator that indicates that a given variable is a pointer. A variable that stores the address of another variable is called a pointer. A pointer can be used to directly access the variable it points to using the indirection operator.

[0075] [Table 7]

[0076] Because each kernel in a particular reconfigurable neural engine 100 implementation resides on the same core (i.e., in the case of a data processing array implementation), such kernels may utilize one or more shared buffers to communicate data with each other. The example source code in Listing 8 illustrates a scenario in which the virtual machine 102 may be hard-coded to implement a particular data flow. In the example of Listing 8, the virtual machine 102 is hard-coded to provide output from the GEMM kernel, which is stored in an accumulation buffer, as input to the BiasAdd kernel. The example of Listing 8 can be used, for example, when instructions are implemented or formatted as static instructions.

[0077] [Table 8]

[0078] When dynamic instructions are used, data flow can be specified at runtime and therefore does not need to be hard-coded within the virtual machine 102. Listing 9 illustrates an exemplary dynamic instruction that specifies data flow that is implemented at runtime through execution of instructions by the virtual machine 102. In this example, the GEMM kernel outputs data to an accumulation buffer, and the BiasAdd kernel receives the accumulation buffer and buffer A as inputs.

[0079] [Table 9]

[0080] In one or more exemplary implementations, data stored in a designated output buffer, such as buffer C, may be output from the reconfigurable neural engine 100 as output data 114, while data stored in other buffers, such as an accumulation buffer or a scratchpad buffer, is not output from the reconfigurable neural engine 100. As an illustrative and non-limiting example, a GEMM kernel may iterate four times to complete a large matrix multiplication operation. Each iteration is specified by a different instruction. In that case, the GEMM kernel generates partial results at the end of each iteration. The partial results are used up in the next iteration of the GEMM kernel. The is_init and is_wait configuration bits can be used to configure the GEMM kernel based on which iteration is being performed. When is_init is true, the GEMM kernel is performing its first iteration and initializes the output memory to zero before storing the partial results. In the second and third iterations, the GEMM kernel reuses the partial results, so the is_init bit is set to false. In the final iteration, the is_wait bit is set in the instruction that causes the GEMM kernel to output the resulting data to its final location (eg, as illustrated in the example in Listing 9).

[0081] In another aspect, instructions for a first kernel and a second kernel (e.g., two different kernels) can be merged into the same opcode if the second kernel always executes after the first kernel. Fusing allows the kernels to be fused into a single monolithic kernel, eliminating the need to provide separate instructions for each of the two kernels. This reduces the overhead of the virtual machine 102. An example of fusing instructions for a GEMM kernel with instructions for a BiasAdd kernel is illustrated in the example of Listing 10.

[0082] [Table 10]

[0083] Although the examples described within this disclosure illustrate a single instance (e.g., a single computational tile), it should be understood that reconfigurable neural engine 100 may be scaled so that there are multiple implementations or instances of reconfigurable neural engine 100. Each implementation or instance of reconfigurable neural engine 100 may be implemented within a different computational tile or other circuit, as described in more detail herein below.

[0084] 7 illustrates an example architecture of an IC 700 capable of implementing one or more instances of a reconfigurable neural engine 100. The IC 700 is an example of an adaptive system. The IC 700 is also an example of a programmable IC. The IC 700 is also an example of a system-on-chip (SoC).

[0085] In the example of Figure 7, IC 700 is implemented on a single die provided within a single integrated package. In other examples, IC 700 may be implemented using multiple interconnected dies, with the various programmable circuit resources illustrated in Figure 7 being implemented across different interconnected dies.

[0086] The IC 700 includes a data processing array 702, a programmable circuit 704, a processor system 706, a network-on-chip (NoC) 708, and one or more hardwired circuit blocks 712. It should be understood that the architecture of the IC 700 is provided for purposes of illustration and not limitation. An IC for use with the inventive configurations described herein may include certain subsystems described herein alone (e.g., individually) or any of various combinations of the subsystems described herein. For example, the IC may include one or more processors. In another example, the IC may include one or more processors (e.g., processor system 706) in combination with the programmable circuit 704. In another example, the IC may include one or more processors (e.g., processor system 706) in combination with a processor array, such as the data processing array 702. In other examples, the IC 700 may include only the data processing array 702 or only the programmable circuit 704.

[0087] The data processing array 702 is implemented as a plurality of interconnected programmable computational tiles 716. As used herein with reference to FIG. 7, the term "tile" refers to a circuit block. The computational tiles 716 may be arranged in an array and are hardwired. Each computational tile 716 may include one or more cores 718 and a memory circuit (abbreviated as "memory" in FIG. 7) 720. In one aspect, each core 718 may execute program code stored in a core-specific program memory included within each respective core (illustrated in FIG. 8). In one aspect, each core 718 may be implemented as one or more processors capable of executing computer-readable instructions.

[0088] In one or more examples, each core 718 can directly access memory circuits 720 within the same computational tile 716 and any other computational tiles 716 adjacent to the core 718 of the computational tile 716 above, below, left, and / or right. For example, core 718-5 can directly read from and / or write to memory circuits 720-5, 720-8, 720-6, and 720-2 (e.g., via respective memory interfaces not shown). Core 718-5 views each of memory circuits 720-5, 720-8, 720-6, and 720-2 as a unified region of memory (e.g., as part of local memory accessible to core 718-5). This facilitates data sharing between different computational tiles 716 within the data processing array 702. In other examples, core 718-5 may be directly connected to memory circuits 720 in other computational tiles.

[0089] The computational tiles 716 are interconnected by programmable interconnect circuitry. The programmable interconnect circuitry may include one or more distinct and independent networks. For example, the programmable interconnect circuitry may include a streaming network formed of streaming connections (shaded arrows) and a memory-mapped network formed of memory-mapped connections (unshaded arrows).

[0090] Cores 718 may be directly connected to adjacent cores 718 via inter-core cascade connections. In one aspect, the inter-core cascade connections are unidirectional direct connections between cores 718, as shown. In another aspect, the inter-core cascade connections are bidirectional direct connections between cores 718. In general, inter-core cascade connections generally allow results stored in accumulation registers of a source core 718 to be provided directly to inputs of a target or load core 718 without passing through a programmable interconnect and / or being written by the first core 718 to a memory circuit 720 so that it can be read by a different core 718.

[0091] In an exemplary embodiment, the computational tiles 716 do not include cache memory. By omitting cache memory, the data processing array 702 can achieve predictable, e.g., deterministic, performance. Furthermore, significant processing overhead is avoided because coherency does not need to be maintained between cache memories located in different computational tiles 716. In a further example, the cores 718 do not have input interrupts. Thus, the cores 718 can operate uninterrupted. By omitting input interrupts to the cores 718, the data processing array 702 can also achieve predictable, e.g., deterministic, performance.

[0092] In one or more other exemplary implementations, the data processing array 702 may include one or more memory tiles. A memory tile is characterized by including a memory, a direct memory access (DMA) engine, and a stream switch that couples to other stream switches in adjacent tiles. A memory tile may be read from and / or written to by any computational tile in the data processing array 702. A memory tile is further characterized by a lack of computational capability. That is, a computational tile includes a core 718, but not a memory tile. In one aspect, the memory circuit 720 may represent a level 1 cache, while the memory tile represents a level 2 cache for the data processing array 702.

[0093] The interface block 722 acts as an interface connecting the computational tiles 716 to other resources of the IC 700. In the example of FIG. 7, the interface block 722 includes a plurality of interconnected interface tiles 724 organized in a row. The interface tiles 724 are connected so that data can be propagated bidirectionally from one interface tile to another. Each interface tile 724 can act as an interface for the row of computational tiles 716 (and / or memory tiles) immediately above it and can interface such computational tiles 716 (and / or memory tiles) with components and / or subsystems of the IC 700, including, but not limited to, the programmable circuit 704 and / or NoC 708.

[0094] The programmable circuit 704 is circuitry that can be programmed to perform a specified function. The programmable circuit 704 may be implemented as or include programmable logic. As an example, the programmable circuit 704 may be implemented as field programmable gate array-type circuitry. The programmable circuit 704 may include an array of programmable circuit blocks. As defined herein, the term "programmable circuit" refers to circuitry used to build reconfigurable digital circuits. Reconfigurable digital circuits are user-definable. The programmable circuit 704 is formed from many programmable circuit blocks that provide basic functionality. The topology of the programmable circuit 704, unlike hardwired circuits, is highly configurable. Each programmable circuit block of the programmable circuit 704 typically includes programmable elements 726 (e.g., functional elements) and programmable interconnects 742. The programmable interconnects 742 provide the highly configurable topology of the programmable circuit 704. The programmable interconnects 742 can be configured on a wire-by-wire basis to provide connections between the programmable elements 726 of the programmable circuit blocks of the programmable circuit 704, and are configurable on a bit-by-bit basis (e.g., where each wire carries one bit of information), as opposed to connections between the computational tiles 716, which may include multi-bitstream connections capable of supporting, for example, packet-based communications.

[0095] Processor system 706 is implemented as hardwired circuitry fabricated as part of IC 700. Processor system 706 may be implemented as or include any of a variety of different processor (e.g., central processing unit) types, each capable of executing computer-readable instructions. For example, processor system 706 may be implemented as one or more processors. Each processor may include one or more cores. Each processor and / or core may be capable of executing computer-readable instructions. In yet another example, processor system 706 may include one or more processors, cores, modules, coprocessors, I / O interfaces, and / or other resources. Processor system 706 may be implemented using any of a variety of different types of architectures. Exemplary architectures that may be used to implement the processors and / or cores of processor system 706 may include, but are not limited to, Complex Instruction Set Computer (CISC) architecture, Reduced Instruction Set Computer (RISC) architecture, vector processing architecture, or other known architectures. Exemplary processors include, but are not limited to, processors having x86-type architectures (e.g., IA-32, IA-64), Power Architecture, ARM processors, etc. In one or more exemplary implementations, processor system 706 may execute a control program that controls the execution of applications (e.g., one or more kernels) within data processing array 702.

[0096] The NoC 708 is a programmable interconnect network for sharing data between endpoint circuits within the IC 700. The endpoint circuits may be located within the data processing array 702, the programmable circuit 704, the processor system 706, and / or selected hardwired circuit blocks 712. The NoC 708 may include high-speed data paths with dedicated switching. In examples, the NoC 708 includes one or more horizontal paths, one or more vertical paths, or both horizontal and vertical paths. The configuration and number of regions shown in FIG. 7 are merely examples. The NoC 708 is an example of a common infrastructure available within the IC 700 to connect selected components and / or subsystems.

[0097] Within the NoC 708, the nets to be routed through the NoC 708 are unknown until a user circuit design is created for implementation within the IC 700. The NoC 708 can be programmed by loading configuration data into internal configuration registers, which define how elements within the NoC 708, such as switches and interfaces, are configured and pass data from switch to switch and between NoC interfaces to connect endpoint circuits. The NoC 708 is fabricated (e.g., hardwired) as part of the IC 700 and is not physically alterable, but can be programmed to establish connections between various master and slave circuits of the user circuit design. In one aspect, the NoC 708 does not implement application data paths (e.g., routes) within it at power-on, but can provide a default path for loading configuration data to selected other subsystems. However, once configured, the NoC 708 implements data paths or routes between endpoint circuits.

[0098] Hardwired circuit blocks 712 include special-purpose circuit blocks fabricated as part of IC 700. While hardwired, hardwired circuit blocks 712 can be configured to implement one or more different modes of operation by loading configuration data into control registers. Examples of hardwired circuit blocks 712 include input / output (I / O) blocks, transceivers for sending and receiving signals to circuits and / or systems external to IC 700, memory controllers, etc. Examples of various I / O blocks may include single-ended and pseudo-differential I / O. Examples of transceivers may include high-speed differential clock transceivers. Other examples of hardwired circuit blocks 712 include, but are not limited to, cryptographic engines, digital-to-analog converters (DACs), analog-to-digital converters (ADCs), etc. Generally, hardwired circuit blocks 712 are application-specific circuit blocks.

[0099] In one or more other examples, the hardwired circuit block 712 may include RAM, such as one or more banks of RAM. By way of example, the RAM may be double data rate (DDR) synchronous dynamic RAM. In yet another example, the hardware circuit block 712 may include high-bandwidth memory (HBM).

[0100] 7 may be initially programmed as part of the boot process of IC 700. During run-time, the programmable circuit resources may be reconfigured. At any point during run-time, all or part of IC 700 may be reconfigured. In some cases, processor system 706 or another processing system disposed within IC 700 may configure and / or reconfigure programmable logic 704 and / or NoC 708.

[0101] Figure 8 illustrates an exemplary implementation of a computational tile 716 of Figure 7. The example of Figure 8 is provided to illustrate certain architectural features of the computational tile 716 and not as a limitation on the configuration of the data processing array 702. Some connections between components and / or tiles have been omitted for ease of illustration.

[0102] In this example, each computational tile 716 includes a core 802, a random-access memory (RAM) 804, a stream switch 806, and a memory-mapped (MM) switch 808. The core 802 includes a scalar processor 820, a vector processor 836, and a program memory 822. The computational tile 716 may further include a set of control registers (not shown) that may be written by the memory-mapped switch 808 to control the operation of various components included in the computational tile 716. The program memory 822 may be loaded with one or more sets of computer-readable (e.g., executable) instructions called “kernels” and virtual machines 102, for example, by loading configuration data. The computational tiles 716 perform data processing operations and are capable of manipulating large amounts of data through the execution of kernels.

[0103] Each core 802 may be directly connected to a RAM 804 located in the same computational tile 716 through a memory interface 832 (e.g., local memory interface 832-1). Within this disclosure, a memory interface such as 832-1 is referred to as a “local memory interface” because the memory interface is used by circuits within the same tile to access the RAM. In comparison, memory interfaces 832-2, 832-3, and 832-4 are referred to as “adjacent memory interfaces” because such memory interfaces are used by circuits in other adjacent tiles to access the RAM. Thus, each core 802 can access the RAM 804 within the same computational tile 716. The cores 802 (e.g., the scalar processor 820 and / or the vector processor 836) can execute program code stored in the program memory 822. The RAM 804 is configured to store application data. The RAM 804 may be updated or written to via the MM switch 808 and / or the stream switch 806. Although not illustrated, the MM switch 808 may be coupled to any memory, including control registers, located within the computational tile 716 so that such memory may be read from or written to. Each of the RAMs 804 may include a direct memory access (DMA) circuit 834 that can read and write data to the RAMs 804 located within the same computational tile 716. The DMA circuit 834 may receive data via the stream switch 806, store such data in the RAMs 804, and / or output data read from the RAMs 804 via the stream switch 806.

[0104] Each core 802 may be directly connected to RAMs 804 located in adjacent computational tiles 716 (e.g., to the north, south, east, and / or west) via the memory interface. Thus, a core 802 may directly access such other adjacent RAMs 804 without initiating a read or write transaction via the stream switch 806 and / or without using DMA circuitry, in the same manner as a core 802 may access a RAM 804 located in the same computational tile 716. As an illustrative example, a core 802 in a computational tile 716-5 may read and / or write to RAMs 804 located in computational tiles 716-5, 716-2, 716-4, and 716-6 without issuing a read or write transaction via the stream switch 806. However, it should be understood that a core may initiate read and write transactions via the stream switch 806 to RAMs 804 of any other computational tiles 716 and / or memory tiles.

[0105] A core 802 may also have direct connections, called cascade connections (not shown), to other cores 802 located in adjacent computational tiles (e.g., to the north, south, east, and / or west), which enable direct sharing of data stored in internal registers (e.g., accumulation registers) of the core 802 (e.g., of the processor 820 and / or the vector processor 836) with other cores 802 without first writing such data to RAM 804 and / or transporting such data through the stream switch 806.

[0106] 7 and 8, one or more implementations of the reconfigurable neural engine 100 may be implemented in the IC 700. In one or more exemplary implementations, one or more instances and / or implementations of the reconfigurable neural engine 100 may be implemented in different ones of the computation tiles 716. For example, different kernels and virtual machines 102 may be stored in the program memory 822. The various buffers illustrated in the examples of FIGS. 4, 5, and / or 6 may be stored in the RAM 804. The scalar processor 820 may execute the virtual machine 102, and the vector processor 836 executes the various kernels under the control of the virtual machine 102 as executed by the scalar processor 820.

[0107] In another exemplary implementation, one or more instances and / or implementations of reconfigurable neural engine 100 may be implemented entirely within programmable circuitry 704. Virtual machine 102 may be implemented within programmable circuitry 704 as a state machine (e.g., within a circuit) or as a soft processor (that executes computer-readable instructions), and kernels coupled to virtual machine 102 are also implemented within programmable circuitry 704.

[0108] In another exemplary implementation, the processor system 706 may execute one or more instances of a virtual machine 102 (e.g., implemented as computer-readable instructions) that couple to respective kernels implemented within the programmable circuitry 704.

[0109] In another exemplary implementation, the processor system 106 may execute one or more instances and / or implementations of the reconfigurable neural engine 100.

[0110] In one or more exemplary implementations, the entity that provides instructions to each reconfigurable neural engine 100 instance and / or implementation may be implemented as a controller within programmable circuit 704, as a processor in processor system 706, or as an external processor (e.g., of a data processing system communicatively linked to IC 700).

[0111] 9 illustrates an example method 900 of operation of a reconfigurable neural engine 100 that includes a virtual machine 102. The virtual machine 102 may run on electronic hardware as described within this disclosure. The method 900 illustrates the run-time operation of the reconfigurable neural engine 100.

[0112] At block 902, the virtual machine 102 receives instructions. More specifically, the virtual machine 102 receives instructions from an entity that implements a higher-level framework. If the reconfigurable neural engine 100 is implemented within a compute tile 716, the instructions may be received via a stream connection.

[0113] In block 904, the virtual machine 102 may analyze the instruction to determine a selected (e.g., first) kernel from multiple kernels coupled to the virtual machine, the multiple kernels being part of the reconfigurable neural engine 100.

[0114] At block 906, the virtual machine optionally parses the instruction to determine (e.g., extract) configuration data from the instruction. The virtual machine may also parse the instruction to determine one or more buffers to be used by the selected kernel. For example, the instruction may specify one or more buffers that store data to be used as input to the selected kernel to perform the operation. The instruction may also specify a buffer in which the selected kernel stores generated data. In one aspect, block 906 reflects operations that may be performed by the virtual machine 102 in response to receiving an instruction having a dynamic instruction format as illustrated in the example of FIG. 3.

[0115] In one or more other exemplary implementations, the virtual machine 102 may be pre-programmed (e.g., hard-coded) with configuration data to be used for the selected kernel and / or specific buffers to be used to provide input and / or store output for the selected kernel. In this example, the received instructions may have a static format, such as illustrated in the example of FIG.

[0116] At block 908, the virtual machine 102 configures the selected kernel using the configuration data to perform the operation specified in the instruction. As described, the configuration data may specify one or more buffers containing input data to the first kernel and a buffer for storing data generated by the first kernel.

[0117] In block 910, the virtual machine 102 executes the selected kernel to perform the operation as configured.

[0118] After block 910, method 900 may loop back to block 902 to continue processing data. In one or more exemplary implementations, one or more buffers used by a selected kernel may be reused for use by a different kernel when the selected kernel finishes using the buffer. As long as data from the reused memory (e.g., a region of memory) is no longer needed by the executing kernel, it may be reused before the kernel completes execution.

[0119] Thus, in one or more examples, the virtual machine 102 can receive and analyze further (e.g., second) instructions to determine a different (e.g., second) kernel from the plurality of kernels. The virtual machine 102 can configure the different kernel to share data with the selected kernel by using at least one selected buffer that was also used by the selected kernel.

[0120] For example, the at least one selected buffer may be one or more buffers configured to store data used as input to the first kernel or a buffer configured to store results generated by the first kernel.

[0121] In another example, a different (eg, second) kernel is configured to obtain input data from at least one selected buffer.

[0122] In one or more examples, the first instruction and the second instruction form a fused instruction.

[0123] In one or more examples, the virtual machine 102 can analyze a different (e.g., second) instruction to extract configuration data from the second instruction and determine from the second instruction at least one buffer to use with the second instruction.

[0124] The virtual machine 102 may execute a different (eg, second) kernel to perform operations as configured.

[0125] While the present disclosure concludes with claims defining novel features, it is believed that the various features described within this disclosure will be better understood by considering the description in conjunction with the drawings. The processes, machines, manufacture, and any variations thereof described herein are provided for illustrative purposes. The specific structural and functional details described within this disclosure should not be construed as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to variously employ the described features in virtually any appropriately detailed structure. Furthermore, the terms and phrases used within this disclosure are not intended to be limiting, but rather to provide an understandable description of the described features.

[0126] For simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding, similar, or like features.

[0127] As defined herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0128] As defined herein, the term "approximately" means that a value or amount is close but not exact; that is, approximately correct or nearly exact. For example, the term "approximately" can mean that a recited characteristic, parameter, or value is within a given amount of the exact characteristic, parameter, or value.

[0129] As defined herein, the terms "at least one," "one or more," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation unless expressly stated otherwise. For example, each of the phrases "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" means A alone, B alone, C alone, A and B in combination, A and C in combination, B and C in combination, or A, B, and C in combination.

[0130] As defined herein, the term "automatically" means without human intervention. As defined herein, the term "user" means a human being.

[0131] As defined herein, the term "computer-readable storage medium" means a storage medium that contains or stores computer-readable instructions (e.g., program code) for use by or in connection with an instruction execution system, apparatus, or device. As defined herein, a "computer-readable storage medium" is not itself a transitory, propagating signal. The various forms of memory described herein are examples of computer-readable storage media. A non-exhaustive list of exemplary computer-readable storage media includes electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, RAM, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electronically erasable programmable read-only memory (EEPROM), static random-access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, and the like.

[0132] As defined herein, the word "if" means "when" or "upon" or "in response to" or "in response to," depending on the context. Thus, the phrase "when it is determined" or "when [a described condition or event] is detected" may be interpreted to mean "upon determining" or "in response to determining," or "upon detecting [a described condition or event]" or "in response to detecting [a described condition or event]," or "in response to detecting [a described condition or event]," depending on the context.

[0133] As defined herein, the term "in response to" and similar phrases as explained above, such as "when," "when," or "upon," means to readily respond or react to an action or event. The response or reaction is performed automatically. Thus, when a second action is performed "in response to" a first action, there is a causal relationship between the occurrence of the first action and the occurrence of the second action. The term "in response to" indicates a causal relationship.

[0134] As defined herein, the term "soft" with respect to a circuit means that the circuit is implemented in programmable logic or programmable circuitry. Thus, a "soft processor" refers to at least one circuit implemented in programmable circuitry that is capable of executing instructions embodied as program instructions.

[0135] As defined herein, the term "substantially" means that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including, for example, tolerances, measurement errors, measurement accuracy limitations, and other factors known to those skilled in the art, may occur in amounts that do not eliminate the effect the characteristic is intended to provide.

[0136] Terms such as first, second, etc. may be used herein to describe various elements, and these elements should not be limited by these terms, as these terms are only used to distinguish one element from another, unless otherwise stated or the context clearly indicates otherwise.

[0137] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems and methods according to various aspects of the disclosed configurations. In some alternative implementations, the actions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. In other examples, the blocks may be executed generally in ascending numerical order, while in still other examples, one or more blocks may be executed in a different order, with the results stored and utilized in a subsequent block or in another block that does not immediately follow. It should also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by dedicated hardware-based systems that perform the specified functions or acts, or a combination of dedicated hardware and computer instructions.

Claims

1. 1. An integrated circuit comprising: Multiple kernels and a virtual machine coupled to the plurality of kernels; 1. An integrated circuit, wherein the virtual machine is configured to interpret instructions directed to different kernels of the plurality of kernels and to control operation of the different kernels of the plurality of kernels in response to the instructions.

2. the virtual machine is executed by a first processor; The integrated circuit of claim 1 , wherein the plurality of kernels are executed by a second processor.

3. the first processor is a scalar processor located within a computational tile of a data processing array; 3. The integrated circuit of claim 2, wherein the second processor is a vector processor located within the computational tile of the data processing array.

4. the virtual machine is implemented in programmable logic; The integrated circuit of claim 1 , wherein one or more of the plurality of kernels are implemented in the programmable logic.

5. The integrated circuit of claim 1 , wherein each kernel of the plurality of kernels is configured to perform a machine learning function.

6. The integrated circuit of claim 1 , wherein the virtual machine uses a static instruction format.

7. The integrated circuit of claim 6 , wherein the virtual machine manages local memory used by the multiple kernels.

8. The integrated circuit of claim 1 , wherein the virtual machine uses a dynamic instruction format.

9. The integrated circuit of claim 8 , wherein the virtual machine manages local memory used by the multiple kernels in response to the instructions.

10. For the selected instruction, the virtual machine:

2. The integrated circuit of claim 1, configured to analyze the selected instruction to determine a selected kernel from the plurality of kernels to which the selected instruction is directed.

11. The integrated circuit of claim 10 , wherein the virtual machine is configured to analyze the selected instructions to determine configuration data for the selected kernel.

12. 11. The integrated circuit of claim 10, wherein the virtual machine is configured to analyze the selected instruction to determine at least one of a buffer containing input data or a buffer in which data generated by the selected kernel is stored.

13. 1. A method comprising: receiving, by a virtual machine running on electronic hardware, a first instruction; using the virtual machine to analyze the first instruction to determine a first kernel from a plurality of kernels coupled to the virtual machine; configuring the first kernel with configuration data to perform an operation specified by the first instruction using the virtual machine; the configuration data specifies a buffer containing input data to the first kernel and a buffer for storing data generated by the first kernel; The method further comprises executing the first kernel by the virtual machine to perform the operation as configured.

14. Parsing the first instruction to extract the configuration data from the first instruction; 14. The method of claim 13, further comprising: analyzing the first instruction to determine one or more buffers to be used by the first kernel.

15. receiving and parsing a second instruction to determine a second kernel from the plurality of kernels; 14. The method of claim 13, further comprising: configuring the second kernel to share data with the first kernel by using at least one selected buffer that was also used by the first kernel.

Images