Reconfigurable neural engine with scalable instruction set architecture

Abstract

An integrated circuit includes a plurality of cores and a virtual machine coupled to the plurality of cores. The virtual machine is configured to interpret instructions directed to different cores of the plurality of cores. The virtual machine is configured to control operation of different cores of the plurality of cores in response to the instruction.

Description

[0001] Reservation of rights for copyrighted material

[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files and records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0003] The present disclosure relates to artificial neural networks, and more particularly to a reconfigurable neural engine with a scalable instruction set architecture for implementing artificial neural networks in integrated circuits. Background Art

[0004] Deep learning refers to a subset of machine learning. To accomplish a given task, deep learning makes use of artificial neural networks. Artificial neural networks, or simply "neural networks," include multiple layers of interconnected nodes that can be used to classify input data into categories of interest. Typically, a neural network will include an input layer and an output layer, and optionally one or more hidden layers between the input and output layers. Neural networks denoted as machine learning models used in deep learning applications typically include many hidden layers, which gave rise to the term "deep neural networks."

[0005] Machine learning models can be developed that specify a particular neural network. Inference refers to computing the output of a machine learning model given input data. Due to the high degree of computational intensity involved in performing inference, a great deal of development effort has been invested in executing machine learning models with greater efficiency on different hardware platforms (e.g., central processing units, graphics processing units, field programmable gate arrays, digital signal processors, and application-specific integrated circuits).

[0006] In some cases, machine learning models are implemented by spatially partitioning fixed-function kernels onto available hardware resources of a hardware platform. Generally speaking, a kernel refers to a design unit that performs an operation in a machine learning model. For example, a kernel can perform functions such as general matrix multiplication, aggregation, pooling, or activation. Spatially partitioning the kernels onto the hardware platform consumes a lot of time, and once completed, it may still result in placing and routing the fixed-function kernels onto the hardware platform, resulting in performance bottlenecks.

[0007] In other cases, developers create hand-tuned kernels that are spatially partitioned onto the available hardware resources of the hardware platform. Creating hand-tuned kernels is also a time-consuming and error-prone task. Hand-tuned kernels may not be compatible with other machine learning models and are often coded at a low level to be able to efficiently utilize the hardware resources of the underlying hardware platform. This makes hand-tuned kernels difficult to scale. Summary of the invention

[0008] In one or more exemplary implementations, an integrated circuit includes a plurality of cores and a virtual machine coupled to the plurality of cores. The virtual machine is configured to interpret instructions directed to different cores of the plurality of cores. The virtual machine is configured to control the operation of the different cores of the plurality of cores in response to the instructions.

[0009] In one or more exemplary implementations, a method includes receiving a first instruction by a virtual machine running on electronic hardware. The method includes parsing the first instruction using the virtual machine to determine a first kernel from a plurality of kernels coupled to the virtual machine. The method includes configuring the first kernel using configuration data by the virtual machine to perform an operation specified by the first instruction. The configuration data specifies a buffer containing input data of the first kernel and a buffer storing data generated by the first kernel. The method includes causing the first kernel to perform the configured operation using the virtual machine.

[0010] This summary is provided only to introduce certain concepts and is not intended to identify any key or essential features of the claimed subject matter. Other features of the arrangements of the present invention will be apparent from the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The arrangement of the present invention is shown by way of example in the accompanying drawings. However, the accompanying drawings should not be interpreted as limiting the arrangement of the present invention to only the specific implementation shown. Various aspects and advantages will become apparent when viewing the following detailed description and when referring to the accompanying drawings.

[0012] Figure 1 An example of a reconfigurable neural engine is shown.

[0013] Figure 2 An example of a static instruction format that can be used by a virtual machine of a reconfigurable neural engine is shown.

[0014] Figure 3 An example of a dynamic instruction format that can be used by a virtual machine of a reconfigurable neural engine is shown.

[0015] Figure 4 An exemplary memory layout of an example for use with a reconfigurable neural engine is shown.

[0016] Figure 5 Another example of a memory layout for an instance of a reconfigurable neural engine is shown, where memory management is performed during runtime.

[0017] Figure 6 Another example of a memory layout for an instance of a reconfigurable neural engine is shown, where memory management is performed during runtime.

[0018] Figure 7 Examples of integrated circuits capable of implementing one or more instances of a reconfigurable neural engine are shown.

[0019] Figure 8 An example of a compute slice of a data processing array capable of implementing an instance of a reconfigurable neural engine is shown.

[0020] Fig. 9 An exemplary method of operation of a reconfigurable neural engine is shown. DETAILED DESCRIPTION

[0021] The present disclosure relates to artificial neural networks, and more particularly to a reconfigurable neural engine with an extensible instruction set architecture for implementing artificial neural networks in integrated circuits. According to the arrangements of the invention described in the present disclosure, the reconfigurable neural engine provides an architectural standard for integrating and / or using kernels. The reconfigurable neural engine can be used by a high-level compiler and / or a machine learning framework to design, control, and run kernels capable of executing machine learning operations of a machine learning model on a selected hardware platform.

[0022] In one or more examples, the reconfigurable neural engine can interact with multiple kernels. Kernels can be function-independent and implement different machine learning functions. A machine learning function is a function performed by a type of layer of a machine learning model implemented as a multi-layer artificial neural network. Examples of different types of machine learning functions that can be implemented as kernels include convolution, general matrix multiplication (GEMM), BiasAdd (adding bias to values), aggregation, pooling, normalization, requantization, and activation functions such as rectified linear units (ReLu), Softmax, hyperbolic tangent (tan h), identity functions, etc.

[0023] Using a reconfigurable neural engine, multiple kernels can be assembled to run within a given hardware resource of a hardware platform. In one aspect, the reconfigurable neural engine operates as glue logic, providing a standardized framework to control the kernels, which are coupled to a virtual machine implemented as part of the reconfigurable neural engine. The virtual machine provides a standardized interface for any kernel coupled to it. The standardized interface provides flexibility because kernels can be added and removed (e.g., swapped) without disrupting the entire software stack of the reconfigurable neural engine.

[0024] The reconfigurable neural engine is capable of running kernels independently of each other. The reconfigurable neural engine facilitates efficient use of the underlying hardware resources of the hardware platform while also providing software-level programmability of the kernel. The reconfigurable neural engine is capable of providing improved computing power and improved memory utilization compared to other kernel implementation technologies. The reconfigurable neural engine implements a control mechanism that allows any high-level framework to program and control the kernels individually. Further aspects of the arrangement of the present invention are described below with reference to the accompanying drawings.

[0025] Figure 1 An example of a reconfigurable neural engine 100 is shown. The reconfigurable neural engine 100 refers to a specific implementation or instance of a virtual machine 102 coupled to one or more cores 104 (e.g., 104-1, 104-2 to 104-N). The virtual machine 102 can receive input data 106, input data 108, and one or more instructions via an instruction stream 110. In an exemplary implementation, each of the input data 106, 108 can represent a data window of a particular size. For purposes of illustration and not limitation, the input data 106 and the input data 108 can each be a 2x8 kB data window (e.g., in this example, the window is double buffered).

[0026] The reconfigurable neural engine 100 can perform memory management of the cores 104 by regulating access to different buffers in the buffers 112 (e.g., 112-1, 112-2, to 112-N) by different cores in the cores 104. The buffers 112 can be local memory buffers of the cores 104. The cores 104 can operate on the input data 106 and / or 108 stored in the buffers 112 and generate results that are output as output data 114. The output data 114 can be an output window or a data stream.

[0027] exist Figure 1 In the example of , kernel 104 can implement any of a variety of different machine learning functions. For example, kernel 104-1 can be a GEMM kernel. Kernel 104-2 can be a BiasADD kernel. Another kernel 104 can be a ReLU kernel. Yet another kernel 104 can be a requantization kernel (e.g., a kernel capable of performing shift and scale functions). Other kernels 104 can implement other machine learning functions, such as Gaussian error linear units (GELU), layer normalization, Softmax, etc. The exemplary machine learning functions provided herein are intended as a non-exhaustive list of machine learning functions that can be implemented by kernel 104.

[0028] The virtual machine 102 can receive instructions (e.g., as an instruction stream 110) from another circuit and / or system. The virtual machine 102 can interpret the received instructions. In response to the instructions and their interpretation, the virtual machine 102 can control the operation of the core 104. The interpretation of the instructions by the virtual machine 102 makes the core 104 programmable. More specifically, in response to the received instructions, the virtual machine 102 can select one or more of the cores 104, configure certain aspects of the selected core 104, manage the local memory buffers of the selected core 104, run the selected core 104, and possibly return data generated by the selected core 104 (e.g., as output data 114).

[0029] In one or more examples, the virtual machine 102 is implemented as a state machine that is configured to receive and operate on an instruction stream 110 generated and / or provided by another circuit and / or system. The virtual machine 102 can parse the incoming instructions to determine the size of the instruction packet, determine which core will run based on the opcode parsed from the instruction, determine memory availability and reuse, and allocate buffers 112 to the core specified to run by the opcode. In this regard, the reconfigurable neural engine 100 is compatible with any high-level framework (e.g., circuit and / or system) that can support the instruction format used by the reconfigurable neural engine 100, and more specifically the virtual machine 102.

[0030] The use of a reconfigurable neural engine 100 provides a higher level of abstraction than conventional techniques for running kernels. This abstraction allows the kernels 104 used with the reconfigurable neural engine 100 to scale more easily and solves placement and routing issues associated with using fixed-function kernels. The reconfigurable neural engine 100 provides standardized interfaces for the kernels 104 that encourage a plug-and-play development paradigm, thereby also supporting kernel reuse. That is, the kernel 104-1 can be replaced with another kernel 104 that performs a different function without requiring significant reimplementation of the software stack, as long as the new or different kernel is able to interface with the virtual machine 102, for example following the standard interface provided by the virtual machine for the kernel.

[0031] The reconfigurable neural engine 100 also provides a high degree of portability between different kernels. For example, another circuit and / or system can be updated to generate compatible instructions for the virtual machine 102. The virtual machine 102 can interpret the received instructions to control, adjust (e.g., configure), and run any kernel coupled to it. However, the underlying implementation details of the kernel can remain unchanged.

[0032] Figure 1Examples may be implemented using any of a variety of different hardware platforms. In one aspect, the hardware platform is implemented as a data processing array. An example of a data processing array is implemented as multiple hardwired circuit blocks. The multiple circuit blocks may be programmable. The data processing array may include multiple computing slices and interface blocks. Generally speaking, a computing slice includes a core with data processing capabilities and a local (L1 cache) memory that can be used to implement buffer 112. Subject to constraints, the local memory may also be accessible to other cores in the data processing array. The data processing array may include a shared memory (L2 cache, also referred to as a memory slice) that is accessible by all cores in the data processing array. In combination Figure 7 An exemplary data processing array is described. Figure 7 The data processing array shown can be programmed to implement one or more operations of a deep neural network.

[0033] In one aspect, the reconfigurable neural engine 100 can be implemented in a computing slice of a data processing array. For example, the reconfigurable neural engine 100 can be deployed to the core of a computing slice of a data processing array. On the other hand, multiple instances of the reconfigurable neural engine 100 can be deployed to multiple cores of a data processing array (e.g., cores of multiple computing slices). In the case where the reconfigurable neural engine 100 runs on multiple computing slices, another circuit and / or system (e.g., a circuit and / or system capable of running a high-level framework such as a controller and / or processor) can be used to implement scheduling instructions and data for multiple computing slices.

[0034] In another aspect, the hardware platform may be implemented as a programmable circuit. The reconfigurable neural engine 100 may be implemented in a programmable circuit. The programmable circuit may include or be implemented as programmable logic. Figure 7 Examples of programmable circuits are described. In still other examples, the reconfigurable neural engine 100 can be implemented as a combination of a processor (eg, a hard processor or a soft processor) and a programmable circuit that implements a core.

[0035] exist Figure 7In the example of, a single IC is able to provide multiple different types of hardware platforms that can implement the reconfigurable neural engine 100. It should be understood that the arrangements of the present invention can be used with other types of ICs, including data processing arrays only, data processing arrays combined with one or more other systems, programmable circuits only, and / or programmable circuits combined with one or more other systems. In addition, the machine learning design can be implemented with one or more instances of the reconfigurable neural engine 100 implemented in one type of hardware platform, while one or more other instances of the reconfigurable neural engine 100 are implemented in different types of hardware platforms. In still other examples, the reconfigurable neural engine 100 can be implemented using a single processor or using one or more processors. In addition, different specific implementations can operate in a coordinated manner as part of the same or common machine learning design implemented in the IC.

[0036] In one or more examples, two (or more) different implementations of the reconfigurable neural engine 100 may be implemented. For purposes of illustration, in an example, a compute slice may perform two (or more) different implementations of the reconfigurable neural engine 100. A first implementation of the reconfigurable neural engine 100 may include kernels A, B, and C configured to operate on mixed precision data. A second implementation of the reconfigurable neural engine 100 may include a kernel configured to operate on integer data (e.g., int8). For example, the kernel in the second implementation of the reconfigurable neural engine 100 may perform matrix multiplication, BiasADD, ReLU, etc. The second implementation of the reconfigurable neural engine 100 may provide improved latency and throughput compared to the first implementation of the reconfigurable neural engine 100 that is capable of operating on mixed precision data.

[0037] The foregoing examples using multiple different reconfigurable neural engine 100 implementations illustrate one manner in which the reconfigurable neural engine 100 can support evolving computational algorithms and data flow throughputs.

[0038] As described above, each core in the reconfigurable neural engine 100 can operate independently under the control of the virtual machine 102 in response to instructions provided to the virtual machine 102. The instructions can be generated by an entity such as another circuit and / or system (e.g., a controller and / or processor) that implements a high-level framework, such as the Tensor Virtual Machine (TVM), PyTorch, TensorFlow, etc. These instructions can be passed down from this entity to various reconfigurable neural engine 100 specific implementations and / or instances during runtime.

[0039] The instructions received and interpreted by the virtual machine 102 may include configuration data, which may be used by the virtual machine 102 to configure the corresponding kernel 104. That is, the virtual machine 102 may receive an instruction. Based on the instruction, the virtual machine 102 may determine the specific kernel 104 to which the instruction belongs, configure the kernel according to predetermined rules within the virtual machine 102 and / or the configuration data specified by the instruction, and call or run the kernel. Configuration data, whether hard-coded within the virtual machine 102 or obtained from the received instruction, may be provided to the kernel specified by the instruction to cause the kernel to operate in a specific operating mode, which may be one of a plurality of different operating modes of the selected kernel 104.

[0040] As an example, consider the case where kernel 104-1 implements a linear approximation kernel. That is, kernel 104-1 can implement any of a plurality of different functions (e.g., an activation function) through linear approximation. In this case, virtual machine 102 determines configuration data specified by a received instruction specifying a linear approximation kernel, provides the configuration data to the kernel, and calls the kernel. In this example, the configuration data may specify different coefficients to be used by the kernel when performing an operation. The specific coefficients may be specific to the operating mode and the linear approximation performed. Therefore, a specific operating mode of a kernel can be invoked by passing different instructions targeting the same kernel to virtual machine 102, albeit with different configuration data.

[0041] In one aspect, the reconfigurable neural engine 100 can be instantiated in a machine learning application specified as a graph. More specifically, the reconfigurable neural engine 100 can be instantiated in an application graph class in a separate header file, similar to how a kernel is instantiated. The header file can use the available application programming interface (API) of the development tool to connect the reconfigurable neural engine 100 (e.g., virtual machine 102) to other kernels and provide data flow information. The standardized interface of the kernel provided by the virtual machine 102 allows the kernel to be reused in different machine learning models. In addition, the kernel can be reconfigured through high-level instructions received by the virtual machine 102.

[0042] The exemplary source code of Listing 1 shows an example of instantiating a reconfigurable neural engine 100 through a graphical application programming interface (API). In the example of Listing 1, the GMIO port is used to transfer data between a device memory (e.g., a double data rate, synchronous dynamic random access memory (DDR)) and the reconfigurable neural engine 100. Device memory refers to global memory accessible by an instance of the reconfigurable neural engine 100 implemented in a hardware platform. GMIO is a class that represents an I / O port property specification for making an external memory mapped connection to or from a global memory (e.g., device memory). In the exemplary source code of Listing 1, "arne" represents a reconfigurable neural engine 100.

[0043] List 1

[0044]

[0045] As shown in Listing 1, the reconfigurable neural engine 100 can support any kernel data access type, including window-based data access through input_window and output_window shown in Listing 1. In addition, the reconfigurable neural engine 100 can support stream-based access, such as input_stream and output_stream in Listing 1. The data access type supported by the reconfigurable neural engine 100 (e.g., virtual machine 102) can include any data type access supported by the kernel.

[0046] The exemplary source code of Listing 2 shows a specific implementation of the reconfigurable neural engine 100 , which includes a stream instruction, two input windows for providing input data 106 and 108 , and an output window for providing output data 114 .

[0047] List 2

[0048] void arne_implementation(input_window_int8*bufA, input_window_int8*bufB,

[0049] input_stream_int8*bufl, output_window_int8*bufOut);

[0050] On the other hand, the reconfigurable neural engine 100 is capable of using multiple stream connections shared on a single physical channel. A packet switching construct can be used to separate multiple stream connections. The exemplary source code of Listing 3 shows how to use the "pktsplit" construct to split multiple input stream connections sharing a single physical channel into instruction windows and data windows. The pktsplit construct in the example of Listing 3 divides a buffer into an instruction buffer and a data buffer for different inputs to the virtual machine 102.

[0051] List 3

[0052] pktsplit<Qm+1> sp_;

[0053] connect<stream,pktstream> (in_[0].out[0], sp_.in[0]);

[0054] connect <pktstream,window<instr_window_sz> >(sp_.out[0], kernels_.in[2]);

[0055] connect <pktstream,window<window0_sz> >(sp_.out[1], kernels_.in[0]);

[0056] The virtual machine 102 may be implemented to support a specific instruction format. In an exemplary implementation, the virtual machine 102 is implemented to use a static instruction format.

[0057] Figure 2 An example of a static instruction format 200 that can be used by virtual machine 102 is shown. The instruction format is called fixed because the instructions have a fixed length. Figure 2 In the example of , instruction format 200 is implemented as a vector including 4 fields, including opcode field 202, configuration field 204, data field 206 (e.g., data 0), and data field 208 (e.g., data 1). In one aspect, each field can be designated as an integer data type of a specific size (such as 8 bits, 16 bits, or 32 bits). It should be understood that the exemplary sizes are provided for illustration purposes and are not intended to be limiting.

[0058] The opcode field 202 specifies the specific kernel to be run. For example, the virtual machine 102 may include a pre-configured mapping of opcodes to kernels. For a given instruction, when parsing the opcode from the instruction, the virtual machine 102 determines the specific kernel 104 to be run in response to the instruction. For example, opcode "0" may be mapped to a general matrix multiplication kernel. Opcode "1" may be mapped to a ReLU kernel. Opcode "2" may be mapped to a BiasAdd kernel, and so on.

[0059] Configuration field 204 may be a customizable field. That is, a kernel designer may define a specific operating mode in which the designer's kernel may operate. Configuration field 204 may include configuration data that specifies a specific operating mode to be implemented by the kernel specified by opcode field 202.

[0060] As an example, the least significant bit (LSB) of configuration field 204 may be used to indicate whether the kernel will output a value for an iteration (e.g., run) of the kernel initiated by the instruction. As another example, the second LSB of configuration field 204 may be used to indicate whether the output of the kernel is to be shifted. If a shift is to be applied, other data bits of configuration field 204 may specify the number of bits of the shifted data.

[0061] Configuration field 204 may be customized on a per-core basis. That is, the configuration data included in configuration field 204 may be core-specific. Thus, the meaning of the bits in configuration field 204 of instructions for core 104-1 may be different than the meaning of the bits in configuration field 204 of instructions for core 104-2. Obviously, virtual machine 102 is able to correctly interpret the configuration bits / data specified in configuration field 204 based on the specific opcode specified in the received instruction.

[0062] The data fields 206, 208 specify the addresses of data (e.g., buffers) that the kernel can use. In one aspect, one or more bits of the configuration field 204 field can specify what each of the data fields 206, 208 will be used for. That is, based on the data bits of the configuration field 204, the data field 206 can specify an input buffer (e.g., input data 106), the data field 208 can specify a different input buffer (e.g., input data 108), and the data field 208 can specify an accumulation buffer, a temporary storage space, and / or an output buffer (e.g., output data 114). Therefore, the kernel designer can specify the purpose of the data specified by one or both of the data fields 206, 208 through the data bits of the configuration field 204. In this example, by using the data fields 206 and / or 208 to specify various types of buffers to be used by the kernel, the static instruction format 200 can be used to implement memory management functions.

[0063] The example source code of Listing 4 can be incorporated into virtual machine 102 and shows an example of virtual machine 102 determining whether the output of the GEMM kernel is to be shifted based on the data bits of configuration field 204. The example of Listing 4 determines whether the output of the GEMM kernel is to be right-shifted and by how many bits.

[0064] List 4

[0065] If (op_code == GEMM) {

[0066] int8 is_init=op_conf&1;

[0067] int8 is_wait=(op_conf>>1)&1;

[0068] int8 is_shift=(op_conf>>2)&1;

[0069] int8 shift_val=is_shift? (int)op_data0:0;

[0070] Figure 2The static instruction format 200 provides certain benefits, such as reducing the complexity of the virtual machine 102 implementation.

[0071] In one or more other examples, the reconfigurable neural engine 100 can be implemented with a virtual machine 102 configured to parse a dynamic instruction format. The dynamic instruction format can be specified as a base class of instructions containing fields shared by all cores in the reconfigurable neural engine 100. Derived classes can specify core-specific configuration fields and data fields.

[0072] The exemplary source code of Listing 5 shows an example of a dynamic instruction header file for a machine learning model that includes GEMM, requantize, loopback, and a general nonlinear kernel. The exemplary source code of Listing 5 shows a base class "op_base". The base class "op_base" contains shared fields such as size_in_bytes (which specifies the size of the instruction in bytes), is_init (which is used for kernel initialization), and is_wait (which is used for kernel control). The base class is instantiated in kernel-specific instructions, derived classes, and custom configuration fields and data fields. For example, the custom data fields is_shift and shift_val are specified for GEMM instructions directed to the GEMM kernel. By providing size_in_bytes through the base class, the virtual machine 102 is able to determine how many bytes are used by the instructions of each kernel during runtime. The shared configuration fields can be decoded by the virtual machine 102 before the custom configuration fields and custom data fields of the instruction are passed to the specified kernel.

[0073] List 5

[0074]

[0075]

[0076] In another aspect, a dynamic instruction format may be used to manage a core's local memory.The dynamic instruction format may specify one or more bits that define an input buffer for a given instruction and / or an output buffer for a given instruction.

[0077] Figure 3 An example of a dynamic instruction format 300 is shown. Figure 3In the example of FIG. 3 , the dynamic instruction format 300 includes a size field 302, an opcode field 304, one or more shared configuration fields 306, one or more custom configuration fields 308, and one or more custom data fields 310. The size field 302 may be a size_in_bytes field. The opcode 304 may specify a specific kernel to be executed, such as in conjunction with Figure 2 For purposes of illustration and not limitation, Figure 3 In the example of , the opcode field 304 may specify kernels such as GEMM, ReLu, BiasAdd, Requantize, etc. The shared configuration field 306 may specify bits indicating the state of is_init, is_out, and / or bits specifying specific input and / or output buffers. The custom configuration field 308 may specify additional configuration fields on a per opcode basis. For example, for an opcode that specifies a requantize kernel, the custom configuration field 308 may specify configuration data such as a multiplier, a Q number, and / or a data shift amount.

[0078] exist Figure 3 In the example of , the kernel can run multiple times before outputting data. For example, if the kernel runs 4 times, the virtual machine 102 will receive four instructions from the kernel. The shared configuration field 306 can contain different data for different iterations or runs of the kernel. The is_init bit can be used in the instruction to indicate whether the specified buffer should be zeroed (e.g., initialized). The is_wait bit can be used in the instruction to indicate whether the kernel is to output data (e.g., in the last iteration) or to store the data in a different buffer for reuse in the next iteration or run (e.g., the first three of four iterations). Other bits can be used to indicate whether the kernel is to shift data and how much to shift for a given run.

[0079] Figure 4 An exemplary memory layout 400 for an example of use with the reconfigurable neural engine 100 is shown. The memory layout 400 can be implemented within a compute slice of a data processing array. For example, the kernel 104 can be loaded into the program memory of the compute slice. More specifically, the kernel 104 can be loaded into the program memory of the core of the compute slice. The remainder of the memory layout 400 can be implemented in the RAM (e.g., the L1 cache) of the compute slice distributed across memory banks 1, 2, 3, and 4. Figure 7 and Figure 8 Examples of compute slices are described.

[0080] For purposes of illustration, in the example of the reconfigurable neural engine 100, kernel 104-1 is a GEMM kernel, kernel 104-2 is a requantization kernel, and kernel 104-3 is a layer normalization kernel. Kernels 104 are loaded into a compute slice program memory and executed therefrom. The example of the reconfigurable neural engine 100 is provided for purposes of illustration and not limitation. Other implementations of the reconfigurable neural engine 100 may include fewer or more kernels and / or different kernels than shown.

[0081] In this example, the shaded portion of the memory layout 400 corresponds to the active or used area of ​​the memory. In this example, the kernel 104-1 is executing and using the shaded area of ​​the memory in the compute slice RAM. Area 402 stores buffer A (e.g., the input matrix of the kernel 104-1), and area 406 stores buffer B (e.g., the input matrix B of the kernel 104-1). Areas 404 and 408 store data for the next run of the kernel 104-1, shown as buffers Ad and Bd, respectively. In this example, each of the memory banks 1 and 2 is divided into two areas so that data can be loaded into buffers A and B in a ping-pong manner, and then loaded into buffers Ad and Bd. Therefore, when the kernel 104-1 operates on buffers A and B, buffers Ad and Bd can be loaded into areas 404 and 408, respectively. For example, buffer A (and buffer Ad) can correspond to input data 106, and buffer B (and buffer Bd) can correspond to input data 108.

[0082] Region 410 stores an accumulation (ACC) buffer for kernel 104-1. Region 412 stores another accumulation buffer (ACCd) for kernel 104-1, which can be used in a ping-pong manner with the ACC buffer of region 410. Region 418 stores buffer C for kernel 104-1, such as an output buffer corresponding to output data 114. Region 414 stores a temporary buffer that can be used for kernel 104-1 during runtime. This temporary buffer may be overwritten by another kernel while kernel 104-3 is running. Region 416 can be used to store a buffer including configuration parameters for a layer normalization kernel (e.g., kernel 104-3). Region 420 can store stack and heap data. Region 422 can be used to store intermediate results as "scratch" data.

[0083] When using the reconfigurable neural engine 100, the cores can be used in a "plug and play" manner. That is, a core can be removed and replaced with another different core (e.g., "swapped out"). The two cores may or may not have the same memory footprint. Therefore, in one aspect, to increase portability and reduce engineering time, the memory management of the reconfigurable neural engine 100 can be determined at creation time.

[0084] Figure 4 The example of shows how to use a static memory layout when all the data required by each kernel fits into the available memory (e.g., fits into the compute slice RAM). Although all kernels can be organized to operate using the available memory, this static memory layout may reduce the reusability of the reconfigurable neural engine 100 for input data of different sizes for different kernels, reduce the portability of the kernels, and reduce the plug-and-play capability of the kernels. Figure 4 As shown in the example of , if one of the cores requires more memory to store configuration parameters, or if an additional core is to be added, the tightly packed data that utilizes all available compute slice RAM for the three cores 104 must undergo significant redesign. However, if sufficient space is available in memory, one or more additional cores can be added to a given reconfigurable neural engine 100 implementation without disturbing the memory layout.

[0085] Figure 5 Another example of a memory layout 500 is shown, where memory management is performed during runtime. Figure 5 In the example of , in response to instructions received by virtual machine 102, virtual machine 102 may perform memory management.

[0086] In one example, the shared configuration field 306 can be used to associate a region of memory with a memory map maintained by the virtual machine 102. For example, a memory map tracks the state of different regions of memory and indicates whether a given region of memory can be reused or repurposed by a different kernel. For example, a (1) for a given memory region in a memory map indicates that the region can be reused. A (0) for a given memory region in a memory map indicates that the region cannot be reused. The memory map can also specify a pointer to the starting location of a buffer stored in a given region and the size of the buffer stored in the region.

[0087] refer to Figure 5 , the shaded areas of memory are currently being used by kernel 104-1, which is also shaded. In this example, kernel 104-1 is running. For purposes of illustration, area 416 does not store layer normalization parameters, but rather other data. Area 414 stores the requantization buffer used by kernel 104-2. Figure 5In the example of , region 402 initially stores buffer A for use by kernel 104-1. Because a memory map maintained by virtual machine 102 indicates that buffer A can be reused (e.g., the memory map includes an entry indicating that buffer A of kernel 104-1 is available for reuse, such as "mem_ready[GEMM_BUFF_A] == 1"), virtual machine 102 has overwritten buffer A in region 402 with layer normalization parameters that kernel 104-3 can use when executing kernel 104-3.

[0088] exist Figure 5 In the example of , in response to determining that kernel 104-1 no longer needs buffer A because the memory map indicates that the state of buffer A is "1", virtual machine 102 loads different data in region 402, effectively reusing region 402 for use by another kernel. In this example, the high-level framework that provides instructions to virtual machine 102 is able to determine when the GEMM operation performed by kernel 104-1 is completed, or when data from region 402 has been read by kernel 104-1 and is no longer needed in memory. The ability to reuse memory regions increases the reusability of memory to support kernels that may need to store various types of data (such as lookup tables) or store large amounts of intermediate results to perform processing. This reuse allows the use of a particular area of ​​memory to be reallocated from a first kernel to a second kernel before the first kernel completes execution, thereby improving runtime efficiency.

[0089] In addition to facilitating memory reusability, the memory management functionality of the virtual machine 102 also allows kernels to share data with each other within the same computing slice. For example, kernels executed by the same kernel (e.g., in the same computing slice) can share input and / or output data with each other. Data sharing between kernels can be achieved by providing instructions to the kernel that cause the kernel to read a specific storage buffer for input and / or store output to a specific (e.g., different) storage buffer.

[0090] The exemplary code of Listing 6 shows a portion of two exemplary instructions having a dynamic instruction format.

[0091] List 6

[0092]

[0093] The first instruction includes an operation code (OP) that selects the GEMM kernel. The first instruction includes an input portion (IN) that specifies the buffer from which the GEMM kernel is to read input data. In this example, the IN portion specifies a value of "1" for input buffers A and B. The output portion (OUT) of the first instruction includes a "1" corresponding to the accumulation buffer, which causes the GEMM kernel to write the output to the accumulation buffer. In response to instruction 1, the virtual machine 102 configures the GEMM kernel to perform operations using the specified buffer and runs the kernel according to the configuration.

[0094] The second instruction includes an operation code (OP) that, in this example, selects the requantization kernel. The second instruction includes a specified input portion (IN) that specifies a buffer from which the requantization kernel is to read input data. In this example, the IN portion specifies a value "1" for the accumulation buffer. The output portion (OUT) of the second instruction includes a "1" corresponding to buffer C, which causes the requantization kernel to write output to buffer C. In response to instruction 2, the virtual machine 102 configures the requantization kernel to perform the configured operation using the specified buffer.

[0095] The example of Listing 6 shows how the virtual machine 102 causes a first kernel (e.g., a GEMM kernel) to keep writing data to a selected buffer (e.g., an accumulation buffer) instead of an output buffer C when processing a received instruction. In the example, the GEMM kernel does not output data, but instead stores the data in another different buffer that can be used by another kernel. Next or subsequently, a second kernel (e.g., a requantization kernel) operates on the selected buffer (e.g., an accumulation buffer) as input data and stores the resulting data in an output buffer C (e.g., output data 114) that can be transmitted to another circuit and / or system via a stream channel. This functionality provides greater control over the memory layout at runtime and provides more efficient use of the memory layout. In addition, when trying different kernel options, the need to redesign the machine learning model by determining a new and / or different memory layout is reduced.

[0096] Figure 6 Another example of a memory layout 500 is shown, where memory management is performed during runtime. Figure 6 In the example of , the memory map indicates that buffer B is available for reuse. That is, because the memory map maintained by virtual machine 102 indicates that buffer B can be reused (e.g., the memory map includes an entry indicating that buffer B of kernel 104-1 is available for reuse, such as "mem_ready[GEMM_BUFF_B] == 1"), virtual machine 102 has overwritten buffer B in region 402 with layer normalization parameters that kernel 104-3 can use when executing kernel 104-3.

[0097] In order to integrate a kernel into a particular reconfigurable neural engine 100 implementation, the kernel must conform to the interface requirements of the virtual machine 102. The virtual machine 102 is solely responsible for interacting with input data and output data on behalf of any integrated kernel. That is, input data is provided to the kernel only through input data 106, 108 and through a stream interface that transmits input data 106, 108, each of which can be an input window of a specific size. The kernel integrated into the reconfigurable neural engine 100 implementation must use pointers to access local memory (e.g., Figure 4 , Figure 5 and / or Figure 6 The contents of the computing chip RAM).

[0098] The exemplary source code of Listing 7 shows a scenario where a kernel uses a pointer to access local memory. In the example of Listing 7, each kernel (e.g., ReLU, BiasAdd, and GEMM) uses T* to access the input buffer and output buffer. As we all know, "*" is the dereference operator, indicating that the given variable is a pointer. A variable that stores the address of another variable is called a pointer. A pointer can be used to directly access the variable pointed to by the pointer using the dereference operator.

[0099] List 7

[0100] template<size_t n,typename T>

[0101] void relu(T*_restrict in, T*_restrict out);

[0102] template <unsigned m0,unsigned n0,unsigned tm,unsigned tn,typename TA,typename TB,typename TC>

[0103] _attribute_((noinline))

[0104] void bias_add(TA*_restrict lhs, TB*_restrict rht, TC*_restrict out);

[0105] template <unsigned m0,unsigned k0,unsigned n0,unsigned tm,unsigned tk,unsigned tn,unsigned tdm,unsigned tdn,typenam TR>

[0106] _attribute_((noinline))

[0107] void mmul_8b_double_buffer_acc_store(const v32int8*_restrict pA, const v64int8*_restrict pB, TR*acc_buf, v32int8*_restrict out, const int is_init, ocnst int shift_val);

[0108] Since each kernel in a particular reconfigurable neural engine 100 implementation resides in the same kernel (i.e., in the case of a data processing array implementation), such kernels can utilize one or more shared buffers to pass data to each other. The exemplary source code of Listing 8 shows a scenario in which the virtual machine 102 can be hard-coded to implement a particular data flow. In the example of Listing 8, the virtual machine 102 is hard-coded to provide the output from the GEMM kernel stored in the accumulation buffer as input to the BiasAdd kernel. For example, the example of Listing 8 can be used where the instructions are implemented or formatted as static instructions.

[0109] List 8

[0110]

[0111] Where dynamic instructions are used, data flow can be specified at runtime and therefore need not be hard-coded in virtual machine 102. Listing 9 shows an exemplary dynamic instruction that specifies data flow that is implemented as runtime by execution of instructions by virtual machine 102. In the example, the GEMM kernel outputs data to the accumulation buffer, while the BiasAdd kernel takes the accumulation buffer and buffer A as input.

[0112] List 9

[0113]

[0114] In one or more exemplary implementations, data stored in a designated output buffer such as buffer C may be output from the reconfigurable neural engine 100 as output data 114, while data stored in other buffers such as an accumulation buffer or a scratch pad buffer is not output from the reconfigurable neural engine 100. As an illustrative, non-limiting example, a GEMM kernel may iterate four times to complete a large matrix multiplication operation. Each iteration is specified by a different instruction. In this case, the GEMM kernel generates a partial result at the end of each iteration. The partial result is used by the next iteration of the GEMM kernel. The is_init and is_wait configuration bits may be used to configure the GEMM kernel according to the iteration being executed. When is_init is true, the GEMM kernel executes the first iteration and initializes the output memory to zero before storing the partial result. In the second and third iterations, the GEMM kernel reuses the partial result, so the is_init bit is set to false. In the last iteration, the is_wait bit is set in the instruction, which causes the GEMM kernel to output the resulting data to the final location (e.g., as shown in the example of Listing 9).

[0115] On the other hand, for those cases where the second kernel is always executed after the first kernel, the instructions for the first kernel and the second kernel (e.g., two different kernels) can be merged into the same opcode. Fusion allows the kernels to be fused into a single monolithic kernel, and separate instructions do not need to be provided for each of the two kernels. This reduces the overhead of the virtual machine 102. The example in Listing 10 shows an example of fusing the instructions of the GEMM kernel with the instructions of the BiasAdd kernel.

[0116] List 10

[0117]

[0118] Although the examples described in this disclosure are shown for a single instance (e.g., a single compute slice), it should be understood that the reconfigurable neural engine 100 can be scaled such that there are multiple implementations or instances of the reconfigurable neural engine 100. Each implementation or instance of the reconfigurable neural engine 100 can be implemented in a different compute slice or in other circuits, as described in more detail below.

[0119] Figure 7 An exemplary architecture of an IC 700 capable of implementing one or more instances of a reconfigurable neural engine 100 is shown. IC 700 is an example of an adaptive system. IC 700 is also an example of a programmable IC. IC 700 is also an example of a system on a chip (SoC).

[0120] exist Figure 7In the example of , IC 700 is implemented on a single die provided in a single integrated package. In other examples, IC 700 can be implemented using multiple interconnected dies, where Figure 7 The various programmable circuit resources shown in are implemented across different interconnected dies.

[0121] IC 700 includes a data processing array 702, a programmable circuit 704, a processor system 706, a network on chip (NoC) 708, and one or more hard-wired circuit blocks 712. It should be understood that the architecture of IC 700 is provided for the purpose of illustration and not limitation. The IC used in conjunction with the arrangement of the present invention described herein may include any of the multiple combinations of independent (e.g., separate) certain subsystems described herein or the subsystems described herein. For example, the IC may include one or more processors. In another example, the IC may include one or more processors (e.g., processor system 706) combined with the programmable circuit 704. In another example, the IC may include one or more processors (e.g., processor system 706) combined with a processor array such as the data processing array 702. In other examples, the IC 700 may include only the data processing array 702 or only the programmable circuit 704.

[0122] The data processing array 702 is implemented as a plurality of interconnected and programmable computing slices 716. Figure 7 The term "slice" is used to refer to a block of circuitry. Computing slices 716 may be arranged in an array and hardwired. Each computing slice 716 may include one or more cores 718 and memory circuits (in Figure 7 In one aspect, each core 718 is capable of executing program code stored in a core-specific program memory contained in each respective core (e.g., Figure 8 In one aspect, each core 718 may be implemented as one or more processors capable of executing computer-readable instructions.

[0123] In one or more examples, each core 718 can directly access the memory circuits 720 within the same computing slice 716 and the memory circuits 720 of any other computing slice 716 adjacent to the core 718 of the computing slice 716 in the up, down, left, and / or right directions. For example, the core 718-5 can directly read and / or write (e.g., via corresponding memory interfaces not shown) the memory circuits 720-5, 720-8, 720-6, and 720-2. The core 718-5 treats each of the memory circuits 720-5, 720-8, 720-6, and 720-2 as a unified memory area (e.g., as part of the local memory accessible to the core 718-5). This facilitates data sharing between different computing slices 716 in the data processing array 702. In other examples, the core 718-5 can be directly connected to the memory circuits 720 in other computing slices.

[0124] The computing slices 716 are interconnected by programmable interconnect circuits. The programmable interconnect circuits may include one or more different and independent networks. For example, the programmable interconnect circuits may include a flow network formed by flow connections (shaded arrows) and a memory mapping network formed by memory mapping connections (unshaded arrows).

[0125] The cores 718 may be directly connected to adjacent cores 718 via core-to-core cascade connections. In one aspect, as shown, the core-to-core cascade connections are unidirectional and direct connections between cores 718. In another aspect, the core-to-core cascade connections are bidirectional and direct connections between cores 718. In general, the core-to-core cascade connections typically allow results stored in accumulation registers of a source core 718 to be provided directly to inputs of a destination core or load core 718 without traversing a programmable interconnect and / or being written by a first core 718 to a memory circuit 720 to be read by a different core 718.

[0126] In an exemplary implementation, the computing slice 716 does not include a cache memory. By omitting the cache memory, the data processing 702 can achieve predictable (e.g., deterministic) performance. In addition, since there is no need to maintain consistency between cache memories located in different computing slices 716, significant processing overhead is avoided. In another example, the core 718 has no input interrupts. Therefore, the core 718 can operate uninterruptedly. Omitting input interrupts to the core 718 also allows the data processing array 702 to achieve predictable (e.g., deterministic) performance.

[0127] In one or more other exemplary implementations, the data processing array 702 may include one or more memory slices (not shown). The memory slices are characterized by including memory, a direct memory access (DMA) engine, and a flow switch coupled to other flow switches in adjacent slices. The memory slices can be read and / or written by any computing slice in the data processing array 702. Another characteristic of the memory slices is the lack of computing power. That is, although the computing slices include the core 718, the memory slices do not. In one aspect, for the data processing array 702, the memory circuit 720 can represent a first-level cache, while the memory blocks represent a second-level cache.

[0128] Interface block 722 serves as an interface to connect compute slice 716 to other resources of IC 700. Figure 7 In the example of , the interface block 722 includes a plurality of interconnected interface slices 724 organized in a row. The interface slices 724 are connected so that data can be transmitted bidirectionally from one interface slice to another interface slice. Each interface slice 724 can serve as an interface for a column of computing slices 716 (and / or memory slices) directly located thereon, and can interface such computing slices 716 (and / or memory slices) with components and / or subsystems of the IC 700, including but not limited to the programmable circuit 704 and / or the NoC 708.

[0129] Programmable circuit 704 is a circuit that can be programmed to perform a specific function. Programmable circuit 704 can be implemented as or include programmable logic. For example, programmable circuit 704 can be implemented as a circuit of field programmable gate array type. Programmable circuit 704 can include an array of programmable circuit blocks. As defined herein, the term "programmable circuit" refers to a circuit system for building a reconfigurable digital circuit. Reconfigurable digital circuits can be user-defined. Programmable circuit 704 is formed by many programmable circuit blocks that provide basic functions. Unlike hard-wired circuits, the topology of programmable circuit 704 is highly configurable. Each programmable circuit block of programmable circuit 704 typically includes a programmable element 726 (e.g., a functional element) and a programmable interconnect 742. Programmable interconnect 742 provides a highly configurable topology of programmable circuit 704. The programmable interconnect 742 may be configured on a per-conductor basis to provide connectivity between programmable elements 726 of programmable circuit blocks of the programmable circuit 704, and may be configured on a per-bit basis (e.g., where each conductor carries a single bit of information), as opposed to, for example, connectivity between compute slices 716 which may include multi-bit stream connections capable of supporting packet-based communications.

[0130] Processor system 706 is implemented as a hardwired circuit manufactured as a part of IC 700. Processor system 706 can be implemented as or include any of a variety of different processor (e.g., central processing unit) types, each type capable of executing computer-readable instructions. For example, processor system 706 can be implemented as one or more processors. Each processor can include one or more cores. Each processor and / or core can execute computer-readable instructions. In another example, processor system 706 can include one or more processors, cores, modules, coprocessors, I / O interfaces and / or other resources. Processor system 706 can be implemented using any of a variety of different types of architectures. Exemplary architectures that can be used to implement processors and / or cores of processor system 706 can include, but are not limited to, complex instruction set computer (CISC) architectures, reduced instruction set computer (RISC) architectures, vector processing architectures, or other known architectures. Exemplary processors include, but are not limited to, processors with x86-type architectures (IA-32, IA-64, etc.), Power Architecture, ARM processors, etc. In one or more exemplary implementations, processor system 706 may execute a control program that controls the execution of applications (eg, one or more cores) within data processing array 702 .

[0131] NoC 708 is a programmable interconnect network for sharing data between endpoint circuits in IC 700. The endpoint circuits may be provided in data processing array 702, programmable circuit 704, processor system 706, and / or selected hardwired circuit blocks 712. NoC 708 may include high-speed data paths with dedicated switches. In an example, NoC 708 includes one or more horizontal paths, one or more vertical paths, or both horizontal and vertical paths. Figure 7 The arrangement and number of regions shown in are examples only. NoC 708 is an example of a common infrastructure within IC 700 that can be used to connect selected components and / or subsystems.

[0132] In NoC 708, the network to be routed through NoC 708 is unknown until the user circuit design for implementation in IC 700 is created. NoC 708 can be programmed by loading configuration data into internal configuration registers, which define how to configure and operate the elements (such as switches and interfaces) in NoC 708 to pass data from switch to switch and between NoC interfaces for connecting endpoint circuits. NoC 708 is manufactured as part of IC 700 (e.g., hard-wired), and although physically unmodifiable, it can be programmed to establish connectivity between different master circuits and different slave circuits of the user circuit design. In one aspect, NoC 708 does not implement any application data path (e.g., routing) therein when powered on, but can provide a default path for loading configuration data into selected other subsystems. However, once configured, NoC 708 implements data paths or routing between endpoint circuits.

[0133] The hardwired circuit block 712 includes a dedicated circuit block that is manufactured as part of the IC 700. Although hardwired, the hardwired circuit block 712 can be configured by loading configuration data into a control register to implement one or more different operating modes. Examples of hardwired circuit blocks 712 may include input / output (I / O) blocks, transceivers for sending and receiving signals to and from circuits and / or systems, memory controllers, etc. outside the IC 700. Examples of different I / O blocks may include single-ended and pseudo-differential I / O. Examples of transceivers may include high-speed differential clock transceivers. Other examples of hardwired circuit blocks 712 include, but are not limited to, cryptographic engines, digital-to-analog converters (DACs), analog-to-digital converters (ADCs), etc. In general, hardwired circuit blocks 712 are dedicated circuit blocks.

[0134] In one or more other examples, the hardwired circuit block 712 may include RAM, such as one or more banks of RAM. For example, the RAM may be a double data rate (DDR) synchronous dynamic RAM. In still another example, the hardware circuit block 712 may include a high bandwidth memory (HBM).

[0135] Figure 7 The various programmable circuit resources shown in 700 may be initially programmed as part of a boot process for IC 700. During operation, the programmable circuit resources may be reconfigured. At any time during runtime, all or a portion of IC 700 may be reconfigured. In some cases, processor system 706 or another processing system disposed in IC 700 may configure and / or reconfigure programmable logic 704 and / or NoC 708.

[0136] Figure 8Shows Figure 7 An exemplary implementation of the computing slice 716 is provided. Figure 8 The example of is intended to illustrate certain architectural features of the computing slice 716 and is not intended to be limiting on the form of the data processing array 702. For ease of illustration, some connections between components and / or slices are omitted.

[0137] In this example, each compute slice 716 includes a core 802, a random access memory (RAM) 804, a stream switch 806, and a memory mapping (MM) switch 808. The core 802 includes a scalar processor 820, a vector processor 836, and a program memory 822. The compute slice 716 may further include a set of control registers (not shown) that may be written by the memory mapping switch 808 to control the operation of various components included in the compute slice 716. The program memory 822 may load one or more sets of computer-readable (e.g., executable) instructions, referred to as "kernels," and the virtual machine 102, such as by loading configuration data. The compute slice 716 is capable of performing data processing operations and operating on large amounts of data through the execution of the kernels.

[0138] Each core 802 can be directly connected to the RAM 804 located in the same computing slice 716 through a memory interface 832 (e.g., a local memory interface 832-1). In the present disclosure, a memory interface such as 832-1 is referred to as a "local memory interface" because the memory interface is used by the circuit in the same slice to access the RAM. In contrast, memory interfaces 832-2, 832-3, and 832-4 are referred to as "adjacent memory interfaces" because such memory interfaces are used by circuits in another adjacent slice to access the RAM. In this way, each core 802 can access the RAM 804 in the same computing slice 716. The kernel 802 (e.g., a scalar processor 820 and / or a vector processor 836) can execute program code stored in the program memory 822. The RAM 804 is configured to store application data. The RAM 804 can be updated or written via the MM switch 808 and / or the stream switch 806. Although not shown, the MM switch 808 can be coupled to any memory located in the computing slice 716, including control registers, so that such memory can be read or written. Each of the RAMs 804 may include a direct memory access (DMA) circuit 834 that is capable of reading and writing data to a RAM 804 located in the same compute slice 716. The DMA circuit 834 may receive data via the stream switch 806 and store such data in the RAM 804, and / or output data read from the RAM 804 through the stream switch 806.

[0139] Each core 802 can be directly connected to RAM 804 located in an adjacent compute slice 716 (e.g., in the north, south, east, and / or west directions). In this way, the core 802 can directly access such other adjacent RAM 804 in the same manner as the core 802 can access the RAM 804 located in the same compute slice 716, without initiating read or write transactions through the stream switch 806 and / or without using DMA circuits. As an illustrative example, the core 802 of the compute slice 716-5 can read and / or write to the RAM 804 located in the compute slices 716-5, 716-2, 716-4, and 716-6 without submitting read or write transactions through the stream switch 806. However, it should be understood that the core can initiate read and write transactions to the RAM 804 of any other compute slice 716 and / or memory slice via the stream switch 806.

[0140] Core 802 may also have direct connections (referred to as cascade connections (not shown)) to other cores in core 802 located in adjacent compute slices (e.g., in the north, south, east, and / or west directions), which allows data stored in internal registers (e.g., accumulator registers) of core 802 (e.g., processor 820 and / or vector processor 836) to be directly shared with other cores 802 without first writing such data to RAM 804 and / or transmitting such data through stream switch 806.

[0141] refer to Figure 7 and Figure 8 In one or more exemplary implementations, one or more reconfigurable neural engine 100 implementations may be implemented in IC 700. In one or more exemplary implementations, one or more instances and / or implementations of reconfigurable neural engine 100 may be implemented in different compute slices 716. For example, different kernels and virtual machines 102 may be stored in program memory 822. Figure 4 , Figure 5 and / or Figure 6 The various buffers shown in the example of can be stored in the RAM 804. The scalar processor 820 can execute the virtual machine 102, and the vector processor 836 executes various kernels under the control of the virtual machine 102 executed by the scalar processor 820.

[0142] In another exemplary implementation, one or more reconfigurable neural engine 100 instances and / or implementations may be implemented entirely in programmable circuitry 704. Virtual machine 102 may be implemented in programmable circuitry 704 (e.g., in a circuit) as a state machine or soft processor (executing computer-readable instructions), while a core coupled to virtual machine 102 is also implemented in programmable circuitry 704.

[0143] In another exemplary implementation, processor system 706 may execute one or more instances of virtual machine 102 (eg, implemented as computer-readable instructions) coupled to a corresponding core implemented in programmable circuitry 704 .

[0144] In another exemplary implementation, the processor system 106 may execute one or more instances and / or implementations of the reconfigurable neural engine 100 .

[0145] In one or more exemplary implementations, the entity providing instructions to each reconfigurable neural engine 100 instance and / or implementation may be implemented as a controller in programmable circuit 704, a processor of processor system 706, or an external processor (e.g., a processor of a data processing system communicatively linked to IC 700).

[0146] Fig. 9 An exemplary method 900 of operation of a reconfigurable neural engine 100 including a virtual machine 102 is shown. The virtual machine 102 may be run on electronic hardware as described in the present disclosure. The method 900 shows the runtime operation of the reconfigurable neural engine 100.

[0147] At block 902, the virtual machine 102 receives instructions. More specifically, the virtual machine 102 receives instructions from an entity implementing a higher-level framework. In the case where the reconfigurable neural engine 100 is implemented in a compute slice 716, the instructions may be received through a stream connection.

[0148] At block 904 , the virtual machine 102 can parse the instruction to determine a selected (eg, first) core from a plurality of cores coupled to the virtual machine. The plurality of cores are part of the reconfigurable neural engine 100 .

[0149] At block 906, the virtual machine optionally parses the instruction to determine (e.g., extract) configuration data from the instruction. The virtual machine may also parse the instruction to determine one or more buffers to be used by the selected kernel. For example, the instruction may specify one or more buffers to store data to be used as input to the selected kernel to perform an operation. The instruction may also specify a buffer in which the selected kernel stores data generated. In one aspect, block 906 reflects the virtual machine 102 responding to receiving the instruction such as Figure 3 Operations that can be performed by instructions having a dynamic instruction format as shown in the example.

[0150] In one or more other exemplary implementations, the virtual machine 102 may be pre-programmed (e.g., hard-coded) with configuration data to be used for the selected core and / or a specific buffer to be used to provide input to and / or store output for the selected core. In this example, the received instruction may have the following Figure 2 The static format is shown in the example.

[0151] At block 908, the virtual machine 102 configures the selected kernel using the configuration data to perform the operations specified by the instruction.As discussed, the configuration data may specify one or more buffers containing input data for the first kernel and buffers storing data generated by the first kernel.

[0152] At block 910 , the virtual machine 102 runs the selected kernel to perform the configured operations.

[0153] After block 910, method 900 may loop back to block 902 to continue processing data. In one or more exemplary implementations, one or more buffers used by the selected kernel may be reused for use by a different kernel in the event that the selected kernel finishes using the buffer. Reuse may be performed before the kernel completes execution as long as the executing kernel no longer needs data from the reused memory (e.g., memory region).

[0154] Thus, in one or more examples, virtual machine 102 can receive and parse additional (e.g., second) instructions to determine a different (e.g., second) core from the plurality of cores. Virtual machine 102 can configure the different core to share data with the selected core by using at least one selected buffer also used by the selected core.

[0155] For example, the at least one selected buffer may be one or more buffers configured to store data used as input to the first kernel, or a buffer configured to store results generated by the first kernel.

[0156] In another example, a different (eg, second) core is configured to obtain input data from at least one selected buffer.

[0157] In one or more examples, the first instruction and the second instruction form a fused instruction.

[0158] In one or more examples, virtual machine 102 can parse a different (eg, second) instruction to extract configuration data from the second instruction and determine from the second instruction at least one buffer to be used by the second instruction.

[0159] The virtual machine 102 can run a different (eg, a second) kernel to perform configured operations.

[0160] Although the present disclosure ends with claims defining novel features, it is believed that the various features described in this disclosure will be better understood by considering the description in conjunction with the accompanying drawings. The processes, machines, manufactures, and any variations thereof described herein are provided for illustrative purposes. The specific structural and functional details described in this disclosure should not be construed as limiting, but merely as a basis for the claims, and as a representative basis for teaching those skilled in the art to variously employ the features described in virtually any appropriately detailed structure. Furthermore, the terms and phrases used in this disclosure are not intended to be limiting, but rather to provide an understandable description of the features described.

[0161] For the purpose of simplicity and clear illustration, the elements shown in the drawings are not necessarily drawn to scale. For example, for clarity, the size of some of the elements may be enlarged relative to other elements. In addition, reference numerals are repeated among the drawings to indicate corresponding, similar or analogous features where deemed appropriate.

[0162] As defined herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0163] As defined herein, the term "approximately" means nearly correct or exact, a value or amount close but not exact. For example, the term "approximately" may mean that the listed characteristic, parameter or value is within a predetermined amount of the exact characteristic, parameter or value.

[0164] As defined herein, the terms "at least one," "one or more," and "and / or" are open-ended expressions that are both conjunctions and transitional conjunctions in operation unless expressly stated otherwise. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," and "A, B, and / or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0165] As defined herein, the term "automatically" means without human intervention. As defined herein, the term "user" means a human being.

[0166] As defined herein, the term "computer-readable storage medium" refers to a storage medium containing or storing computer-readable instructions (e.g., program code) that are used by or in conjunction with an instruction execution system, device, or apparatus. As defined herein, a "computer-readable storage medium" is not a transient propagation signal itself. Various forms of memory as described herein are examples of computer-readable storage media. An incomplete list of examples of computer-readable storage media includes electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. An incomplete list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, RAM, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, etc.

[0167] As defined herein, the term “if” means “when” or “at” or “in response to” or “responsive to”, depending on the context. Thus, the phrase “if it is determined” or “if [a specified condition or event] is detected” may be understood to mean “upon determination” or “in response to determination” or “upon detection of [a specified condition or event]” or “in response to detection of [a specified condition or event]” or “responsive to detection of [a specified condition or event]”, depending on the context.

[0168] As defined herein, the term "in response to" and similar language as described above (e.g., "if," "when," or "upon") means to be susceptible to a response or reaction to an action or event. The response or reaction is automatically performed. Thus, if a second action is performed "in response to" a first action, there is a causal relationship between the occurrence of the first action and the occurrence of the second action. The term "in response to" indicates a causal relationship.

[0169] As defined herein, the term "soft" with respect to a circuit means that the circuit is implemented in programmable logic or programmable circuitry. Thus, a "soft processor" refers to at least one circuit implemented in programmable circuitry that is capable of executing instructions embodied as program instructions.

[0170] As defined herein, the term "substantially" means that the characteristics, parameters or values ​​described need not be achieved precisely, but deviations or variations (including, for example, tolerances, measurement errors, measurement precision limitations and other factors known to those skilled in the art) may occur in amounts that do not interfere with the effect that the characteristics are intended to provide.

[0171] The terms first, second, etc. may be used herein to describe various elements. These elements should not be limited by these terms, as these terms are only used to distinguish one element from another element, unless otherwise stated or the context clearly indicates otherwise.

[0172] Flowchart and block diagram in the figure show the possible concrete implementation of the system and method of various aspects of the arrangement of the present invention, functionality and operation.In some alternative concrete implementations, the operation pointed out in the frame may not occur in the order pointed out in the figure.For example, depending on the function involved, two frames shown continuously can be performed substantially simultaneously, or these frames can sometimes be performed in reverse order.In other examples, the frame can be usually performed in increasing numerical order, and in other examples, one or more frames can be performed in a variable order, and its result is stored and used in subsequent frames or frames that do not follow closely thereafter.It will also be noted that the combination of each frame in the block diagram and / or the flow chart illustration and the frame in the block diagram and / or the flow chart illustration can be realized by a system based on special-purpose hardware that performs a specified function or action or performs a combination of special-purpose hardware and computer instructions.

Claims

1. An integrated circuit, comprising: Multiple cores; as well as a virtual machine coupled to the plurality of cores; The virtual machine is configured to interpret instructions directed to different cores among the plurality of cores and control operations of the different cores among the plurality of cores in response to the instructions.

2. The integrated circuit according to claim 1, wherein: The virtual machine is executed by the first processor; and The plurality of kernels are executed by a second processor.

3. The integrated circuit according to claim 2, wherein: The first processor is a scalar processor disposed in a computing slice of a data processing array; and The second processor is a vector processor disposed in the computing slice of the data processing array.

4. The integrated circuit according to claim 1, wherein: The virtual machine is implemented in programmable logic; and One or more of the plurality of cores are implemented in the programmable logic.

5. The integrated circuit according to claim 1, wherein: Each core of the plurality of cores is configured to perform a machine learning function.

6. The integrated circuit of claim 1, wherein: The virtual machine uses a static instruction format.

7. The integrated circuit according to claim 6, wherein: The virtual machine manages local memory used by the plurality of cores.

8. The integrated circuit of claim 1, wherein: The virtual machine uses a dynamic instruction format.

9. The integrated circuit of claim 8, wherein: The virtual machine manages local memory used by the plurality of cores in response to the instructions.

10. The integrated circuit of claim 1, wherein: For the selected instructions, the virtual machine is configured to: The selected instruction is parsed to determine a selected core from the plurality of cores to which the selected instruction points.

11. The integrated circuit of claim 10, wherein: The virtual machine is configured to parse the selected instructions to determine configuration data for the selected kernel.

12. The integrated circuit of claim 10, wherein: The virtual machine is configured to parse the selected instruction to determine at least one of a buffer containing input data or a buffer storing data generated by the selected kernel.

13. A method comprising: Receiving a first instruction by a virtual machine running on electronic hardware; parsing the first instruction using the virtual machine to determine a first core from a plurality of cores coupled to the virtual machine; configuring the first kernel using the virtual machine with configuration data to perform the operation specified by the first instruction; wherein the configuration data specifies a buffer containing input data for the first kernel and a buffer storing data generated by the first kernel; as well as The virtual machine runs the first kernel to perform the configured operation.

14. The method according to claim 13, further comprising: parsing the first instruction to extract the configuration data according to the first instruction; as well as The first instruction is parsed to determine one or more buffers to be used by the first kernel.

15. The method according to claim 13, further comprising: receiving and parsing a second instruction to determine a second kernel from the plurality of kernels; as well as The second core is configured to share data with the first core by using at least one selected buffer also used by the first core.

Images