Method and apparatus for accessing external memory in neural network processing system

Abstract

Artificial intelligence is a field with extremely high computational density, so that the artificial intelligence can be expensive, time-consuming and energy-consuming. It is lucky that many calculations required for artificial intelligence can be executed in parallel so that a dedicated processor can greatly improve computational performance. In particular, artificial intelligence generally requires a large number of matrix operations to implement neural networks such that dedicated matrix processor circuits may improve performance. To perform all of these matrix operations, the matrix processor circuitry must be supplied with data to be processed quickly and efficiently, otherwise the matrix processor circuitry eventually becomes free or spends a large amount of time to load matrix data of different weights. Accordingly, an apparatus and method are provided for efficiently operating an external interface on a neural network processor that efficiently interleaves and overlaps external memory operations along with computing operations such that both the external interface and the matrix processor circuit are efficiently used.

Description

[0001] Related Cases

[0002] The present disclosure is a continuation-in-part of and claims the benefit of priority to U.S. Patent Application No. 17 / 504,488, filed on October 18, 2021, entitled “Method and Apparatus For Efficiently Processing Convolution Neural Network Operations,” which is a continuation-in-part of U.S. Patent Application No. 16 / 568,195, filed on September 11, 2019, entitled “Method and Apparatus For Efficiently Processing Convolution Neural Network Operations.” The entire text of the foregoing disclosure (including all references cited therein) is hereby incorporated herein by reference. Technical Field

[0003] The present invention relates to the field of digital processing circuits. In particular, but not by way of limitation, the present invention discloses digital circuit designs, control systems, and operating modes for managing on-chip and off-chip data access for digital circuits that perform matrix operations. Background Art

[0004] Computer system designers are always trying to design faster and faster computer systems. Faster computer systems allow extremely complex computational models, such as weather prediction, protein folding, celestial mechanics, artificial intelligence, and complex 3D video rendering, to be executed faster. In addition, the computational models being simulated can be made even more detailed, thus showing more accurate results.

[0005] In order to design faster computer systems, many different techniques are used. One of the simplest techniques is to increase the clock speed at which the computer system operates, but due to the physical properties of current transistor materials, increasing clock speeds is becoming much more difficult. Processing wider data structures can also improve computer performance, but this only helps certain types of computing tasks that can utilize wider data structures. Two popular techniques for improving processing speed are parallel processing, such as implementing multiple computing cores within a computer processor and combining thousands of different computer systems on a computer network to collaborate to solve a single computing problem.

[0006] One of the areas where specialized processors are most in demand is in the field of artificial intelligence (AI). AI is increasingly being used for a wide variety of complex tasks, such as image recognition, high performance computing (HPC), scientific computing, machine learning, data mining, speech recognition, and self-driving cars. AI applications often rely heavily on matrix operations from the mathematical field of linear algebra. Specifically, matrix operations are required to implement artificial neural networks (ANNs) that learn from a training data set and then later apply that learning to new input data.

[0007] Artificial intelligence is a very computationally intensive computing field that is in dire need of computational optimization due to the very frequent use of matrix calculations. One of the most popular techniques to improve the performance of artificial intelligence applications is to create specialized digital processing circuits for performing the matrix operations required to implement artificial neural networks. Dedicated matrix processors exploit the inherent parallelism in matrix operations and thus efficiently perform the matrix calculations commonly used within artificial intelligence.

[0008] Artificial intelligence systems perform a huge number of matrix calculations. Matrix calculations performed by artificial intelligence systems are often performed repeatedly using the same set of matrix weights but different data sets. Similarly, a data set can be processed by a series of matrix operations, thereby generating intermediate results and then ending with a final result. Each of these matrix calculations involves moving a large amount of data into and out of a matrix processor from a memory storage device. These memory storage device access operations can consume a lot of power and can reduce computation time. Without proper coordination, these memory storage device accesses can reduce the performance of a dedicated processor. Therefore, it is desirable to further develop new technologies for efficiently organizing and controlling multi-matrix processor circuits in order to optimize computational tasks associated with implementing artificial neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In the drawings, which are not necessarily drawn to scale, similar numerals throughout the several views describe substantially similar components. Similar numerals with different letter suffixes represent different instances of substantially similar components. The drawings generally illustrate various embodiments discussed in this document by way of example and not by way of limitation.

[0010] Figure 1A Conceptual diagram illustrating a single-layer artificial neural network.

[0011] Figure 1B Conceptual diagram illustrating a three-layer artificial neural network.

[0012] Figure 1C Conceptual diagram illustrating a sparse two-layer artificial neural network.

[0013] Figure 2A A block diagram illustrating one embodiment matrix processor circuit that may be used to perform matrix calculations.

[0014] Figure 2B illustrate Figure 2A Conceptual diagram of a matrix processor circuit having a four-by-four weight matrix consisting of sixteen weight value elements W[0,0] to W[3,3] stored within a wide SRAM memory system.

[0015] Figure 2C A block diagram illustrating an abstract matrix processor circuit that may be used to perform matrix calculations.

[0016] Figure 2D Block diagram illustrating an abstract matrix processor circuit with bus interfaces on all sides.

[0017] Figure 3A A block diagram illustrating a matrix processor array used to create a neural processing unit (NPU).

[0018] Figure 3B illustrate Figure 3A An embodiment of a neural processing unit (NPU).

[0019] Figure 4 An example illustrating the general processing of matrix operations.

[0020] Figures 5A to 5C Timing diagram illustrating processing a neural network using both input operations and compute operations to optimize external interface usage.

[0021] Figure 6 A timing diagram illustrating processing of a neural network using an input and output interleaving technique that further optimizes use of external interfaces.

[0022] Figure 7 A flowchart is illustrated that describes how out-of-order processing can provide more computing options to further improve utilization of system resources.

[0023] Fig. 8A A timing diagram illustrating processing of a neural network via out-of-order processing.

[0024] Figure 8B illustrate Fig. 8A Continuation of the timing diagram of , where intermediate data is described as being stored externally and then reloaded into the neural processing unit. DETAILED DESCRIPTION

[0025] The following detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings show descriptions according to example embodiments. These embodiments (which are also referred to herein as "examples") are described in sufficient detail to enable those skilled in the art to practice the invention. It will be appreciated by those skilled in the art that the specific details in the example embodiments may not be required to practice the invention. For example, although some example embodiments are disclosed with reference to specific matrix processor circuit implementations, the disclosed technology can be used with any other implementation of the matrix processor circuit. Without departing from the scope of the claimed content, the example embodiments can be combined, other embodiments can be utilized, or structural changes, logical changes, and electrical changes can be made. Therefore, the following detailed description should not be interpreted in a limiting sense, and the scope is defined by the attached claims and their equivalents.

[0026] In this document, as is common in patent documents, the terms "a" or "an" are used to include one or more than one. In this document, the term "or" is used to refer to a non-exclusive "or", such that "A or B" includes "A but not B", "B but not A", and "A and B", unless otherwise indicated. In addition, the entire text of all publications, patents, and patent documents mentioned in this document are incorporated herein by reference as if individually incorporated by reference. In the event of inconsistent usage between this document and those documents so incorporated by reference, the usage in the (several) incorporated references should be considered supplementary to the usage of this document; for irreconcilable inconsistencies, the usage in this document controls.

[0027] Neural Network Overview

[0028] One of the core technologies in artificial intelligence (AI) is the use of artificial neural networks (ANNs). ANNs first learn from training data and are then later used to make logical inferences based on new input data. ANNs were originally designed to be similar to biological neuronal networks in animal brains.

[0029] Figure 1A A conceptual diagram illustrating a very simple single-layer, four-input artificial neural network 100. Figure 1A, the input data vector (consisting of input data 101 to 104) is provided with training data during a training session, and then provided with new input data when reasoning is performed using the artificial neural network. The input data vector (consisting of input data 101 to 104) is processed using the weight data stored in the weight matrix 120 to create an output data vector (consisting of output data 161 to 164). Many different types of data processing can be performed using the weight matrix 120 (e.g., Hadamard product, Frobenius inner product, matrix addition, etc.). However, this document will focus on the well-known matrix product.

[0030] After processing the input data vector using the weight matrix 120, the system creates an output data vector (consisting of output data 161-164). The output data vector can be combined with an output function 170 to create a final output 191 of the artificial neural network 100. The output function 170 can be referred to as an activation function. During a training session, the output data can be compared to a desired target output (not shown), and the difference between the output data and the desired target output can be used to adjust the weight data within the weight matrix 120 to improve the accuracy of the artificial neural network 100.

[0031] It should be noted that Figure 1A The four-input artificial neural network of illustrates just one example of a simple small artificial neural network 100. Artificial neural networks can be constructed much wider than just four inputs. Multiple independent artificial neural networks can be used in parallel and their outputs can be combined.

[0032] Artificial neural networks can include multiple weight matrix layers so that very complex computational analyses of input data can be performed. For example, Figure 1B A three-layer artificial neural network is illustrated in which an input data vector (consisting of input data 101 to 104) is processed using a first weighting matrix 121 to create a first intermediate data vector (consisting of data 141 to 144 (also referred to herein as intermediate data or intermediate results)). Next, the first intermediate data vector (consisting of data 141 to 144) is processed using a second weighting matrix 122 to create a second intermediate data vector (consisting of data 151 to 154). The second intermediate data vector (consisting of data 151 to 154) is then processed using a third weighting matrix 123 to create an output data vector (consisting of output data 161 to 164). The output data vector (consisting of output data 161 to 164) can then be processed by an output function 170 to create a final output 191. Alternatively (or additionally), the output data vector (consisting of output data 161 to 164) can also be used as intermediate data that is fed into additional artificial neural network layers (not shown) so that a very complex hierarchical artificial neural network can be created.

[0033] It should be noted that not all input data and intermediate data will affect all subsequent intermediate data and output data. Figure 1C illustrates a much sparser example neural network. For example, Figure 1C As illustrated in , only the input data 101 and 102 affect the intermediate data 141 , rather than all the input data ( 101 to 104 ) affecting the intermediate data 141 .

[0034] Example Matrix Processor Circuit

[0035] As reference Figures 1A to 1C As explained, artificial intelligence relies on a large number of very computationally intensive matrix operations to initially learn using training data to adjust the weights in the weight matrix. Later, those adjusted weight matrices are used to perform complex matrix calculations with new input data sets to reason about the new input data. Fortunately, the linear algebra matrix operations used in artificial neural networks allow for many performance optimizations because there is a lot of parallelism in the required matrix calculation tasks. Therefore, many dedicated processors for artificial intelligence applications have been created. These dedicated AI processors can use a single instruction multiple data (SIMD) architecture, where each instruction is used to process wide data vectors so that matrix operations can be performed efficiently.

[0036] In order to provide optimal processing for artificial intelligence tasks, a dedicated matrix processor can be used. A matrix processor is a digital processing circuit that has been designed to help efficiently perform artificial intelligence computing tasks. Specifically, the matrix processor is designed in a way to quickly read input data vectors, output data vectors, and matrix weight data in a parallel format to achieve high throughput. In this way, the matrix processor can be used for forward propagation reasoning as well as back-propagation artificial intelligence learning.

[0037] Figure 2A A block diagram illustrating one embodiment of an example matrix processor circuit 200 handling a data vector having six data elements. It should be noted that the matrix processor circuit can be caused to handle data vectors having more or fewer data elements within each data vector.

[0038] Figure 2A The matrix processor circuit 200 includes a local wide static random access memory (SRAM) bank 230. The wide SRAM bank 230 is configured so that all wide data rows can be accessed in a single memory cycle. In this way, an entire input data vector or an entire row of weight values ​​from a weight matrix can be read from or written to the SRAM bank 230 in a single memory cycle. The matrix processor circuit 200 also includes an operand register file 210 for storing input data vectors and other data vectors that can be used as operands during calculations.

[0039] The wide SRAM bank 230, operand register file 210, and operand bus 221 are coupled to a multiplexer bank 240 that provides operand data to a multiply and accumulate (MAC) unit bank 260. A local control system 205 within the matrix processor circuit 200 controls all of these individual circuit elements to perform the desired data vector processing operations. Thus, the local control system 205 selects between data stored within the wide SRAM 230, data in the operand register file 210, and data on the operand bus 221 to provide to the multiply and accumulate (MAC) unit 260 for data vector processing.

[0040] The computation output results from the multiplication and accumulation (MAC) unit group 260 may be stored in the result register file 250. These output results may be output in parallel in their original form using the result bus 291. Alternatively (or as a supplement to the original output data), the results in the result register file 250 may also be combined with the reduction tree 270 to provide a single output on the reduction bus 295. It should be noted that the reduction tree 270 may be implemented external to the matrix processor circuit 200.

[0041] It should be noted that for some operations, the results stored in the result register file 250 may be used as input operands in subsequent data vector calculations. To handle such calculations, there is a data path from the result register file 250 back to the multiply and accumulate (MAC) unit group 260. The local control system 205 is used to control exactly how the multiply and accumulate (MAC) unit 260 will select input data to be processed and how the multiply and accumulate (MAC) unit 260 will process the input data.

[0042] Figure 2B Conceptually illustrate how a four-by-four weight matrix consisting of elements W[0,0] through W[3,3] may be stored in the wide SRAM bank 230. Note that the weight values ​​in the weight matrix are stored aligned with the row structure of the underlying SRAM 230 memory so that all rows of weight values ​​may be read out of the wide SRAM bank 230 in a single memory cycle. For example, weight values ​​W[0,0], W[0,1], W[0,2], and W[0,3] from the first row in the weight matrix may be read out of the wide SRAM bank 230 in a single memory cycle and provided simultaneously in parallel to individual multiplication and accumulation (MAC) units 260. Other input operands for the computation may be read from the operand register file 210 or the operand bus ( Figure 2B 260 in parallel.

[0043] Figure 2A and 2B The matrix processor circuit 200 is illustrative of only one possible embodiment of a matrix processor circuit. Figure 2A and 2B Details of the matrix processor circuit 200 can be found in U.S. Patent Application Serial No. 16 / 149,054, entitled "Methods and Apparatus for Constructing Digital Circuits for Performing Matrix Operations," which is hereby incorporated by reference. However, the matrix processor circuit can be implemented in many different ways and in many different sizes.

[0044] Abstract matrix processor circuit

[0045] Matrix processor circuits can be implemented in many different sizes and in many different ways. However, to further efficiently process matrix operations, multiple matrix processor circuits can be combined together in an efficient manner so that a controlled network of matrix processor circuits can perform a wide variety of matrix operations. Therefore, in order to simplify this disclosure, reference will be made to Figure 2C Discloses the abstract matrix processor circuit.

[0046] Figure 2C A first block diagram illustrating an example abstract processor circuit 201 (also referred to herein as a matrix processor circuit 201 or matrix processor 201). The abstract matrix processor circuit 201 receives input data on one or more operand buses. Figure 2C In a particular embodiment of , there are two operand buses: operand bus 221T from the top and operand bus 221L from the left. Data received on the operand buses can be used directly by processing logic 267 or can be stored in memory bank 230 for later use. The received data may include weight matrix data, input data operand vectors, or other data. Memory bank 230 may also include a register file that is tightly coupled to processing logic 267. It should be noted that the operand and result buses can be placed on all different sides of the matrix processing array circuit. Figure 2D An example is illustrated having operand buses (221T, 221B, 221L, and 221R) and result buses (291T, 291B, 291L, and 291R) on all sides.

[0047] Return to reference Figure 2C, the matrix processor circuit 201 also receives commands on the command bus 207. The control system 205 within the matrix processor circuit 201 parses the received commands and uses the commands to determine how the processing logic 267 will be used to process the data. The processing logic 267 can be implemented in many different ways, as long as the matrix processor 201 performs the desired matrix operations and outputs the correct matrix operation results. For example, the processing logic 267 can use a single instruction multiple data (SIMD) processor, a digital signal processor (DSP), a conventional central processing unit (CPU) core, such as Figure 2A and 2B 2. The matrix operations may be performed by the highly parallelized, dedicated matrix processor circuit 200 described in detail in conjunction with FIG. 2, or in any other manner that performs the desired matrix operations.

[0048] The abstract matrix processor circuit 201 can be designed to operate using many different types of data formats and data precision levels. For example, the abstract matrix processor circuit 201 can process integers, 16-bit floating point numbers, 32-bit floating point numbers, or any other data format. Many different matrix operations can be implemented in the abstract matrix processor circuit 201. Two well-known matrix operations that can be included are matrix dot products and matrix cross products.

[0049] The control system 205 instructs the processing logic 267 to output the results of the requested matrix operations on one or more result buses 291. In some embodiments, the matrix processor 201 will include reduction logic to output the results in reduced form on the reduction bus 295. As will be described later, the reduction logic may also be implemented external to the matrix processor circuit 201.

[0050] The operand buses 221T and 221L are wide parallel buses so that an entire input data vector can be loaded into the abstract matrix processor circuit 201 in a single cycle. Similarly, all weight matrix rows from the weight matrix can be read into the local memory bank 230 of the abstract matrix processor circuit 201 in a single cycle. Similarly, the result buses 291R and 291B are also wide parallel buses so that all output data vectors can be output from the abstract matrix processor circuit 201 in a single cycle. The local memory bank 230 is a very important component of the abstract matrix processor circuit 201. As stated earlier, the memory bank 230 of the abstract matrix processor circuit 201 is both wide and deep to optimize performance.

[0051] The local memory bank 230 is wide because an entire data vector can be written to or read from the local memory bank 230 in a single cycle. For example, in a large matrix processor circuit 201 handling a 16 by 16 element matrix where each element is a 16-bit floating point value, the local memory bank 230 can read out a 256-bit value so that all sixteen element data vectors of 16-bit data values ​​can each be read from the local memory bank 230 in a single cycle.

[0052] The local memory bank 230 is deep in that it is constructed to be large enough to store multiple different weight matrix sets. In this way, the matrix processor circuit 201 can be used to perform matrix operations for multiple different artificial neural network layers without having to reload different matrix weight values. For example, if the matrix processor circuit 201 is unable to perform an operation for a particular neural network layer because the required input data vector is not yet available, then that matrix processor circuit 201 can be used instead to perform matrix operations for other neural network layers or other neural networks. The deep memory bank 230 allows the matrix processor 201 to be used very efficiently because it can handle a steady stream of requested matrix operations for many different neural networks without even having to load new weight matrix data. Loading weight matrix data can be one of the most time consuming (and energy consuming) tasks for the matrix processor circuit 201.

[0053] In addition to storing the weight values ​​of multiple different weight matrices, local memory group 230 can also be used to store other information that may be needed, such as input data vectors, output data vectors, error vectors, etc. Intermediate result data vectors from forward pass operations can be stored in local memory group 230 and then accessed later when performing related back-propagation operations. Another very important type of data that can be stored in local memory group 230 is the matrix weight gradient. The matrix weight gradient includes an adjustment matrix of the weight matrix that can be periodically used to update the weight matrix.

[0054] Combine matrix processors into arrays

[0055] Figure 2C The abstract matrix processor circuit 201 described in the figure can be used alone to perform simple matrix operations very quickly. For example, the matrix processor circuit 201 can be used to completely process Figure 1A It can also be used to implement the very small artificial neural network described in . It can also be used to perform the required matrix calculations of all three artificial neural network layers 121, 122 and 123 by using it in series. Figure 1B The small three-layer artificial neural network described in .

[0056] However, most artificial neural networks have to deal with Figure 1A and 1B201. Thus, it may be desirable to combine the computational power of many different matrix processor circuits 201 in order to handle wider artificial neural networks and multi-layer artificial neural networks. In this way, much larger multi-layer artificial neural networks that are used to perform useful artificial intelligence tasks may be handled very efficiently.

[0057] Figure 3A A block diagram illustrating a first embodiment of an architecture that uses multiple matrix processor circuits in a coordinated manner to efficiently process wide multi-layer artificial neural networks. Figure 3A In FIG. 1 , each individual matrix processor circuit is labeled “MP” which stands for matrix processor. Figure 3A , the matrix processor circuits are arranged in a grid array format. Between the individual matrix processor circuits of the matrix processor array are bus wiring and combinatorial logic 399, which couples all the individual matrix processor circuits in the array to buffers carrying input data vectors, matrix weight values, and result data vectors. The matrix processor array and bus wiring and combinatorial logic 399 may be referred to as a matrix processor array 397. The bus wiring and combinatorial logic 399 may be implemented in different ways to achieve different goals.

[0058] In one embodiment, to provide input data vectors to the matrix processor array 397, the vector scalar processor (VSP) 371 is coupled to the operand bus of each individual matrix processor circuit in the matrix processor array 397 via bus wiring 399. This can be achieved by coupling the operand bus 221L, such as Figure 2C . In this manner, data vectors may be loaded into individual matrix processor circuits in the matrix processor array 397. The data vectors may include weight matrix rows, input data vectors, or any other required data for a processing operation to be performed. It should be noted that due to the presence of multiple buses, these data vector loading operations may be performed in parallel.

[0059] Similarly, the result bus of each individual matrix processor circuit in the array is coupled to the accumulation buffer (Acc buffer) 375 on the bottom of the matrix processor array 397 using bus wiring and combinatorial logic 399. This can be done by Figure 2C The result bus 291B is coupled to the accumulation buffer 375 (Acc buffer) on the bottom to achieve, as Figure 3A . The accumulation buffer 375 contains both storage for storing result data vectors and processing logic for performing various vector processing operations on received result data vectors. For example, the accumulation buffer 375 can combine partial result data vectors from multiple different matrix processor circuits into a single complete output data vector result.

[0060] All individual matrix processor circuits in the matrix processor array 397 receive commands on their individual command buses. In this way, each individual matrix processor circuit in the array can be individually controlled. For example, individual matrix processor circuits can be informed when data becomes available on their operand buses and what operations are performed. By carefully controlling each individual matrix processor circuit in the matrix processor array 397 in a coordinated manner, the matrix processor array 397 becomes a very powerful system for efficiently processing the matrix operations required for neural network applications. Specifically, the matrix processor array 397 together with all supporting circuit systems (accumulation buffer 375, vector scalar processor 371, direct memory access (DMA) unit 381 (also referred to as direct memory access (DMA) system 381 in this article), etc.) can be referred to as a neural processing unit (NPU) 300.

[0061] Overview of Neural Processing Unit External Data Management

[0062] To handle Figure 3A To process the neural network within the matrix processor array 397, the neural processing unit (NPU) 300 must receive data from an external source. Figure 3A As shown in the figure, there is an input / output (I / O) interface 395 (also referred to as an external interface system 395 or external interface 395) of the neural processing unit (NPU) 300 controlled by direct memory access (DMA) 381, which can receive and send data from outside the neural processing unit (NPU) 300.

[0063] To operate most efficiently, the neural processing unit (NPU) 300 must use its various internal memories and external data sources coupled to the input / output interface 395 as efficiently as possible. Ideally, Figure 3A The neural processing unit (NPU) 300 will typically store the data it needs to perform matrix calculations in the internal local memory as frequently as possible so that processing can proceed uninterrupted. When accessing external data sources through the input / output (I / O) interface 395, such external access operations should be performed as little as possible and as efficiently as possible. Therefore, the neural processing unit (NPU) 300 should try to retain important data that will be reused in the internal memory until it is no longer needed.

[0064] Neural Processing Unit Overview

[0065] refer to Figure 3A, the neural processing unit (NPU) 300 is controlled by a scheduler and sequence processor (SSP) 350. The scheduler and sequence processor (SSP) 350 is responsible for sending all cycle commands to the vector scalar processor (VSP) 371 and all the various individual matrix processor (MP) circuits within the matrix processor array 397.

[0066] The scheduler and sequence processor (SSP) 350 may include a tree traverser (TW) 351 and a row sequencer (RS) 353. The tree traverser (TW) 351 traverses the neural network tree and is responsible for obtaining the data slices required for processing. The row sequencer (RS) 353 may not only handle one row at a time; instead it may combine multiple rows into a single row sequence. The row sequencer (RS) 353 is responsible for implementing all cycle commands for each data slice. In each operation cycle, each vector scalar processor (VSP) 371 follows the cycle command received from the scheduler and sequence processor (SSP) 350. The same is true for the matrix processors, accumulation buffers 375, and direct memory access (DMA) units 381 within the matrix processor array 397. In order for the neural processing unit (NPU) 300 to operate properly, each resource needs to be carefully sequenced. Therefore, a cycle command set needs to be generated for each operation cycle.

[0067] The direct memory access (DMA) unit 381 can issue multiple requests in parallel. This document will use the term direct memory access (DMA) unit, which can be used to access external DRAM memory. However, the DMA unit 381 should be considered a general memory access system that can be used to access any different type of memory storage system (DRAM, SRAM, flash memory) with any type of memory interface (serial memory bus, network access, parallel bus, etc.). Additional information about the DMA unit 381 will be provided in a later section.

[0068] The computer system may use many neural processing units 300 within a computer system focused on artificial intelligence. Different neural processing units may be controlled in some way to cooperate in solving the same neural network problem. (Alternatively, a single neural processing unit may be divided into multiple regions and completely different matrix computation problems may be processed simultaneously within the same neural processing unit.) When several different neural processing units cooperate to solve the same computation problem, the DMA system 381 may be used to transfer data from one neural processing unit to another neural processing unit. This allows different neural processing units to solve different stages or layers of the same neural network computation problem.

[0069] The matrix processor array 397 is responsible for processing convolutional and fully connected (FC) neural network layers. The matrix processor array 397 can also perform grouped convolutions. Within the bus wiring and combinational logic 399 there may be partial summation units that can combine data values ​​on the way to the accumulation buffer 375.

[0070] The accumulation buffer 375 is responsible for accumulating the results from the various matrix processors in the matrix processor array 397. The accumulation buffer 375 may also perform quantization operations. The accumulation buffer 375 may also perform activation functions for the neural network. For example, ReLu (Rectified Linear Unit), PReLu (Parametric ReLU), Leaky ReLU, and other well-known activation functions may be performed within the accumulation buffer 375 circuit system. It should be noted that some activation functions may also be performed in the vector scalar processor (VSP) 371.

[0071] The vector scalar processor (VSP) 371 is responsible for other calculations not performed in the matrix processor array 397 or the accumulation buffer 375. The vector scalar processor (VSP) 371 can perform pooling functions, such as the maximum pooling and average pooling functions commonly used in convolutional neural networks. The vector scalar processor (VSP) 371 can also perform data reshaping functions. For example, data can be changed from one data format to another data format in order to increase or reduce the accuracy of the data. The vector scalar processor (VSP) 371 has its own dedicated VSP memory 372, which is illustrated as the VSP memory block on the left side of the vector scalar processor (VSP) 371.

[0072] Each matrix processor (MP) within the matrix processor array 397 includes its own local memory (as previously described in Figure 2D 371). This local matrix processor memory is primarily used to store the neural network weight matrices. It should be noted that the matrix weights can also be loaded from the vector scalar processor (VSP) 371 in a timely manner; however, to reduce power consumption and memory port bandwidth usage, the matrix weights are typically stored in the local matrix processor memory. This local storage of matrix weights allows those matrix weights to be reused as much as possible. The local matrix processor memory 230 can also be used to store intermediate results.

[0073] Neural Processing Unit Data Access

[0074] Figure 3AThe neural processing unit 300 uses direct memory access (DMA) 381 coupled to input / output port 395 to send and receive data from outside the neural processing unit 300. It should be noted that this document will specifically refer to "direct memory access"; however, the term direct memory access in this document can apply to any data movement into or out of the neural processing unit 300. For example, DMA unit 381 can access off-chip DRAM memory, or it can access memory from a different partition of the same integrated circuit chip.

[0075] In one embodiment, the direct memory access (DMA) system 381 will access a high-speed memory system, such as an external double data rate (DDR) memory system. However, all disclosed techniques related to accessing data external to the neural processing unit 300 are applicable to any type of external interface system and any type of data storage system used to access data. For example, the external interface system 395 of the DMA unit 381 may include a peripheral component interconnect express (PCIe) bus to obtain data from an external source. Similarly, the external interface system 395 may include a mobile industry processor interface (MIPI) bus. Another data bus type of interface is the Advanced eXtensible Interface (AXI) bus commonly used with ARM (Advanced RISC Machine) based processor systems.

[0076] In addition to just data bus systems, the techniques can also be used to access any type of computer network system to access data on any network accessible storage system. For example, a direct memory access (DMA) system 381 can access a well-known Ethernet interface to access data on a server coupled to a computer network. Alternatively, a DMA system can use an on-chip data fabric to access data stored locally on the same chip. For example, as stated earlier, a complex neural processing chip may include multiple different neural processing units 300 that cooperate to perform neural network processing tasks. Therefore, a direct memory access (DMA) system 381 can use an on-chip data fabric to transfer data between different neural processing units 300 implemented on the same chip.

[0077] Direct memory access (DMA) system 381 can operate using all different types of host and slave interface systems. For host type interfaces, the host controls the interface. Therefore, using the host type interface, DMA system 381 can start data transfer at any time. Using the slave type interface, DMA system 381 can only start data transfer when the slave interface receives permission from the host of the interface system. The slave can send an "indication status" message to the host, which notifies the host whether the slave can currently receive or send data. Then, the host of the interface system can notify the slave when the slave can send data, so that the DMA system 381 on the slave interface can then respond with data transfer.

[0078] A host-based external interface system will typically require less data buffering capacity because the host has the ability to initiate data transfers when necessary. A slave interface may require more data buffering because the slave interface lacks control of the data interface and can therefore only transfer data when it receives permission from the slave interface host to transfer data.

[0079] General Management Overview

[0080] For simple convolutional neural networks, current neural network processing systems operate in a very simple and straightforward manner. To describe the conventional neural network processing system operation, we will refer to Figure 4 Rendering Process Figure 1B An example of a three-layer neural network as described in .

[0081] refer to Figure 4 , the first step is to fully load all tensor input data into the neural processing unit. Figure 4 430. When sufficient input tensor data has been loaded into memory, the system may begin processing the tensor data. This is illustrated as the system processing layer 1 431. Note that the computation of layer 1 431 may begin before all the data has been loaded. The neural processing system then processes through all of the convolutional neural network layers, illustrated as layer 2 432 and layer 3 433, until the entire three-layer neural network is fully processed to obtain the final output result. When the output result data is available, the neural processing system may begin to emit the output result data, such as Figure 4 , where output data 470 is sent out on the external interface.

[0082] There are several problems with this conventional system. One of the biggest drawbacks is that this is a huge waste of the input / output bandwidth available on the external I / O interface. Specifically, the system first uses the I / O bandwidth to load the input data 410, but then the system allows the I / O bandwidth of the external interface to be completely idle while processing data through all neural network layers. Only after all matrix calculations for layer 1 431, layer 2 432, and layer 3 433 does the system finally begin to use the external interface again to send out the final result data 470. Therefore, in this system, the I / O bandwidth may need to be over-provisioned as an expensive high-speed interface in order to minimize latency in such a system.

[0083] Figure 4 Another disadvantage of the technique described in is that this is a very high latency solution. Specifically, Figure 4The techniques described in seldom exploit the natural parallelism inherent in computational problems. There is a small amount of parallelism because the processing of layer 1 431 can begin before all input data 410 is received and the resulting output data 470 can begin to be emitted before the processing of the final layer (layer 3 433) is completed. But otherwise, Figure 4 All processing operations performed in the system are done serially and therefore the system has high latency.

[0084] Figure 4 Another disadvantage of the system described in is that the computing circuitry is not utilized efficiently. Initially, the computing circuitry must wait until sufficient data has been loaded into the system before it can begin computing layer 1 431. And once the computation of the three layers is complete, the computing circuitry is idle until additional input data to be processed is loaded.

[0085] Figure 4 A final additional problem with this conventional solution is that this conventional system requires a large amount of buffer memory within the neural processor unit for storing all input data, intermediate results created during neural network processing, and final output data. Therefore, due to all these shortcomings, significant improvements are needed.

[0086] Matrix processor array with improved external data input

[0087] To improve the traditional system, the present invention uses a data loading system that utilizes the very valuable external interface bandwidth more efficiently. In addition, the computing circuits are utilized in a much more efficient manner. The net result of these improvements is a reduction in computing latency and a reduction in the amount of memory required within the network processing unit.

[0088] Figures 5A to 5C An example of how the proposed improved system may operate is described below. Figure 5A , the neural processing unit issues a first data loading operation 511 for the first tensor data subset required for the computation operation. In order to start the computation quickly, the tensor data subset can be read directly into the matrix processing unit that will perform the computation operation. For example, referring to Figure 3A , DMA unit 381 can read a subset of tensor data from external interface 395 directly into a matrix processor (MP) of matrix processor array 397, so that the subset of tensor data is immediately ready for computational operations. After data loading 511 is completed, computational operations 531 can then begin immediately.

[0089] Next, if Figure 5BAs illustrated in FIG. 5 , while executing the computation operation 531, the DMA unit 381 may read in an additional tensor data subset from the external interface 395 via a data load operation 512. Thus, the system simultaneously computes a result (computation operation 531) and loads an additional tensor data subset (data load operation 512). Since the matrix processors in the matrix processor array 397 are currently busy processing the computation operation 531, the second data load operation 512 may load the tensor data subset into a buffer (e.g., a VSP memory within the vector scalar processor 371).

[0090] After the first computational operation 531 has been completed, the neural processing unit 300 may perform a very fast data move (DM) 522 operation to load the newly obtained tensor data subset into the matrix processor array 397 of the neural processing unit 300. This data move (DM) 522 operation is completely local within the neural processing unit 300 and is therefore very fast. Then, a second operation 532 may be started on a second tensor data subset. It should be noted that both the external interface 395 and the matrix processor array 397 are kept very busy, thus maximizing the utilization of the hardware resources in the neural processing unit 300.

[0091] The NPUs can perform highly parallel operations in this way. Figure 5C , while performing the second computation operation 532, the DMA unit 381 may read the third tensor data subset from the external interface 395 through a data loading operation 513. This third data loading operation 513 is performed simultaneously with the second computation operation 532. While the matrix processors in the matrix processor array 397 are busy processing the second computation operation 532, the third data loading operation 513 loads the third tensor data subset into a buffer (e.g., a memory within the vector scalar processor 371). After the second computation operation 532 has been completed, the neural processing unit 300 may perform a very fast data move (DM) 523 operation to load the newly obtained third tensor data subset into the matrix processor array 397 of the neural processing unit 300, and then start another computation operation 533.

[0092] refer to Figures 5A to 5C , it can be seen that both the external interface 395 and the matrix processor array 397 remain busy performing operations. In this way, the disclosed system provides better utilization of hardware resources and reduces latency.

[0093] Matrix processor array with interleaved external data input and output

[0094] Figures 5A to 5C The example of accessing an external data source to input tensor data for processing. However, both data input and output on the external interface can be interleaved to even further improve the utilization of hardware resources.

[0095] Figure 6 A timeline diagram illustrating activity on the external interface 395 and computational operations performed within a matrix processor of the matrix processor array 397 for neural network computations. Initially, a first subset of tensor data is loaded via input operation 611. Once that first subset of tensor data has been loaded, a matrix processor in the matrix processor array may begin computational operation 631.

[0096] While computation operation 631 is being executed, the neural processing unit may load additional data tensor sub-sets. Specifically, input operation 612 and input operation 613 may be executed while the matrix processor is executing computation operation 631, thereby performing efficient parallelization operations.

[0097] When computation operation 631 is complete, the system may then output the result of that computation via output operation 671. At the same time, the matrix processor may perform subsequent computation operation 632, again performing an efficiently parallelized operation.

[0098] While the matrix processor is performing the computation operation 632, the external interface 395 may input additional tensor data subsets. Specifically, the input operation 614 may be performed after the output operation 671 is completed. Thus, it can be seen that the bandwidth of the external interface 395 is used very efficiently with minimal waste.

[0099] When calculation operation 632 is completed, the system cannot immediately output the result of that calculation because input operation 614 is still executing. However, since the data has been loaded, the matrix processor can immediately start calculation operation 633. When input operation 614 is completed, the neural processing unit can then output the data from calculation operation 632 through output operation 672.

[0100] Next, when the computation operation 633 is completed, the system can immediately start the computation operation 634 because the required tensor data subset has been loaded. And when the output operation 672 is completed, the output data from the computation operation 633 can be sent out through the output operation 673. After the output operation 673, the external interface can then continue to input more data through the input operation 614.

[0101] like Figure 6 As can be seen in FIG. 1 , interleaving of input and output operations can efficiently utilize both the external interface and the matrix processor.

[0102] Data Compression

[0103] As from Figure 6As can be seen from the operational examples described in , large amounts of data are constantly being moved in and out of the neural processing unit to perform these large data intensive matrix operations. In order to reduce the amount of memory required to store the data and reduce the amount of external interface bandwidth usage, the system of the present invention can implement compression and decompression of the data.

[0104] Return to reference Figure 6 , the top of the timeline illustrates a set of input and output data operations, where data is input from an external storage device to the neural processing unit or output from the neural processing unit to an external storage device. During any or all of those operations, data can be compressed and decompressed on the fly using logic within the neural processing unit.

[0105] Matrix Processor Array Out-of-Order Operations

[0106] Neural networks are usually processed in a simple and straightforward order through the different layers of the neural network. Specifically, the first layer 1 is fully calculated first, then layer 2 is fully calculated, then layer 3 is fully calculated, and so on, such as Figure 4 However, operating in this manner may miss some of the parallelism inherent in neural networks, which can be used to better optimize the utilization of hardware resources, such as memory storage devices, memory bandwidth, and processing power of the neural network processing unit.

[0107] Can be returned by reference Figure 1C A sparse neural network is used to illustrate the example of parallelism. Figure 1C , the intermediate result 141 depends only on the input data 101 and the input data 102. Therefore, if the neural network processing unit has loaded the input data 101 and the input data 102 in the local memory, but the input data 103 and the input data 104 have not been loaded into the local memory, then that neural network processing unit may continue to calculate the intermediate result 141 before receiving the input data 103 and the input data 104 that constitute the full input tensor data vector. Specifically, the neural processor unit may combine the input data 101 and the input data 102 to calculate the intermediate result 141 while loading the input data 103 and the input data 104 and thereby achieve higher utilization by exploiting parallelism.

[0108] This type of more efficient operation using parallelism can be further achieved by performing computational operations for subsequent neural network layers before fully completing the current neural network layer. In this way, there are more computational options that can be performed so that the system can further optimize the use of resources within the system.

[0109] Figure 7 Basic operations that describe how a neural processing unit can operate to perform additional computations in other layers, if possible. Figure 1CThe example sparse neural network and Fig. 8A The timing diagram is used to describe Figure 7 Initially, at step 710, the system divides the neural network into various processing stages. In this way, the computational problem is divided into individual processing stages that can be more easily handled.

[0110] Next, at step 720, the neural processing unit loads the first subset of tensor data to be processed. (Again, it should be noted that the system can decompress the data on the fly during the load operation.) Return to reference Figure 1C In the example of , this tensor data subset may be only input data 101 and input data 102 to start processing, such as by Fig. 8A Next, at step 730, the neural processing unit then processes as many results, all or part of them, as it can compute using the current subset of tensor data and any saved intermediate data it may have. This is performed by using the dependency information in the neural network. For example, returning to reference Figure 1C , it can be seen that the intermediate results 141 and 142 in the first layer can be calculated using only the input data 101 and the input data 102. Therefore, the neural processing unit can calculate the intermediate results 141 and 142 through processing operations 831 and 832, respectively, as shown in FIG. Fig. 8A While processing operations 831 and 832 are being performed, the external interface may be used for other load or store operations 812, such as Fig. 8A As described in .

[0111] In addition to calculating the intermediate results 141 and 142, the intermediate results 141 and 142 may also be used to calculate the final output 151, as shown by Figure 1C . These dependencies are represented by the connections shown as lines between nodes on one neural processing layer and the next neural processing layer. Therefore, in step 730, the system may also calculate the final output 151 via operation 833, thereby keeping the matrix processor busy because additional data is not yet available. It should be noted that during this calculation, input data 101 and input data 102 may be loaded simultaneously via load operation 813. At this point, the system has calculated everything it can based on the available loaded data and proceeds to the next step.

[0112] Return to reference Figure 7, at step 740, the system saves all input data and partial results needed for further computation, saves any final output results, and discards all other data. In this example, input data 102 is saved because it is needed to compute intermediate result 143; intermediate result 141 is saved because it is needed to compute output 152; intermediate result 142 is saved because it is needed to compute output 153; and final result 151 is saved because it is the final result. However, input data 101 may be discarded because it is no longer needed to compute any other results. Again, this dependency information is from Figure 1C By discarding unnecessary data, the system reduces the amount of memory required.

[0113] Next, at step 750, the system determines whether that data subset is the final data subset. If there is additional data, then the system proceeds to step 753, where the system may store any intermediate or final data to free up memory if necessary. (Note that this storage operation may include compressing the data.) This storage operation is Fig. 8A 814 is illustrated as a load / store operation.

[0114] Next, the system proceeds to step 755 to load another subset of tensor data if needed. (This may involve decompressing the data while loading it.) Fig. 8A In the example illustrated in , the system has loaded input data 103 and input data 104. Using that newly loaded subset of tensor data, the system returns to step 730 to again process as many results, in whole or in part, as it could compute using the current subset of data and any saved data it may have, based on the dependency information inherent in the neural network.

[0115] Referring back to step 750, when the last tensor data subset has been processed, the system proceeds to step 760 to determine if there are additional sub-stages to be processed. If there are additional sub-stages, then the system proceeds to step 770, where the system moves to the next sub-stage for processing. Therefore, the system returns to step 720 to load the tensor data subset for the next sub-stage. It should be noted that the system may have pre-fetched this tensor data subset using the techniques previously described in this document. Using the loaded tensor data subset, the system can then calculate the intermediate result 143 through operation 834, such as Fig. 8A As described in .

[0116] Referring back to step 760, if this is the last sub-stage, then the process is complete.

[0117] By being able to process data out of the strict normal level-by-level order, the system is able to gain more parallelism and therefore better utilize available hardware resources.

[0118] Storing and reloading intermediate data

[0119] Hardware resources (e.g., local memory storage, memory bandwidth, and processing power) are very valuable and therefore care must be taken to use these resources efficiently. As previously stated, the disclosed system discards data that is no longer needed for future calculations to free up local memory. Another technique that can be used to optimize resource utilization is to temporarily move data out of local high-speed memory if it is not immediately needed. In this way, the system frees up more valuable memory for other data. When stored intermediate data is necessary, the stored data can then be reloaded by the system at a later time.

[0120] Return to reference Figure 1C In the example of , the intermediate data 142 is not discarded because it will be needed for later calculations. Specifically, the intermediate data 142 is needed to calculate the output 154, such as Figure 1C However, since the computation of output 154 will not be performed quickly, the system may choose to store intermediate data 142 until it is needed later in order to free up memory. Figure 8B A timeline illustrating an example of this technique. Note that this is a simplified example and in a typical operation there may be thousands of data units and the time between later operations is much longer.

[0121] Figure 8B illustrate Fig. 8A , which begins with the load operation 813 of input data 103 and 104 and the calculation operation of final data 151. At this point, the system may choose to store intermediate data 142 via a store operation 815. This store operation frees up memory for other load and / or store operations 816.

[0122] The system then proceeds with other computations, where intermediate data 143 and 144 are computed by operations 834 and 835, respectively. Next, the final output 152 is computed by operation 836. At this point in this simple example, it is close to the time when the intermediate data 142 will be needed again, and therefore, the system begins a load operation 817 to reload the intermediate data 142 into the local memory. Computations may continue in the matrix processor concurrently with this load operation 817. Specifically, the final output 153 is computed by operation 837 while the intermediate data 142 is being reloaded into the neural processing unit with the external interface using operation 817. As Figure 8BIn the last illustrated step of this example, after the intermediate data 142 is reloaded back into the local memory, the intermediate data is used to compute the final output 154 through a computation operation 838. Further load or store operations 818 may be performed during operation 838 to compute the final output 154. For example, if the final outputs 151, 152, and 153 have not yet been emitted, the system may emit the final outputs 151, 152, and 153. Alternatively, the system may begin loading a subset of data tensors for later operation.

[0123] The aforementioned technical disclosure is intended to be illustrative rather than restrictive. For example, the above-mentioned embodiments (or one or more aspects thereof) can be used in combination with each other. A person skilled in the art will understand other embodiments after reading the above description. Therefore, the scope of the claims should be determined with reference to the appended claims together with the full scope of equivalents authorized by the claims. In the appended claims, the terms "comprising" and "in which" are used as the plain English equivalents of the corresponding terms "including" and "wherein". Moreover, in the appended claims, the terms "comprising" and "including" are open-ended, that is, systems, devices, articles or processes containing elements other than those listed after the term in the claim are still considered to fall within the scope of that claim. In addition, in the appended claims, the terms "first", "second" and "third" etc. are only used as labels and are not intended to impose numerical requirements on their objects.

[0124] The abstract is provided to comply with 37 CFR § 1.72 (b), which requires that the abstract allow the reader to quickly ascertain the nature of the technical disclosure. The abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the above detailed description, various features may be grouped together to simplify the disclosure. This should not be interpreted as an intention that unclaimed disclosed features are essential to any claim. Rather, the subject matter of the invention may lie in less than all the features of a particular disclosed embodiment. Therefore, the appended claims are hereby incorporated into the detailed description, with each claim standing on its own as a separate embodiment.

Claims

1. A method for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, the method comprising the following steps: processing a portion of multi-layer computations of a multi-layer neural network operation in a matrix processor unit to generate a temporary data set to be used in a subsequent stage of the multi-layer neural network operation; While performing the processing, loading from the external repository a subset of tensor data required for additional computation of the multi-layer neural network operation; and A temporary data subset not required for subsequent computations of the multi-layer neural network operation is discarded.

2. The method of claim 1 for efficiently accessing external memory of a neural network processor while executing a multi-layer neural network operation, wherein the subset of tensor data from the external memory repository required for additional computation of the multi-layer neural network operation comprises a subset of tensor input data for a new neural network operation.

3. The method of claim 1 for efficiently accessing external memory of a neural network processor while executing a multi-layer neural network operation, wherein the subset of tensor data from the external memory repository required for additional computation of the multi-layer neural network operation comprises a set of matrix weights for the neural network operation.

4. The method of claim 1 for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, wherein the subset of tensor data from the external memory repository required for additional computation of the multi-layer neural network operation comprises a subset of data previously stored in the external memory repository.

5. The method of claim 1 for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, the method further comprising: A subset of the temporary data set is stored in the external repository as intermediate result tensor data.

6. The method of claim 5 for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, the method further comprising: Reload the intermediate result tensor data from the external repository.

7. The method of claim 1 for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, the method further comprising: The temporary data is stored in the external repository as a final result of the multi-layer neural network operation.

8. A neural network processor device for performing multi-layer neural network calculations, the neural network processor device comprising: An external data interface; the external data interface is coupled to a memory storage device external to the neural network processor device; a vector scalar processor coupled to the external data interface, the vector scalar processor being configured to receive incoming data; a matrix processor circuit array coupled to the vector scalar processor, the matrix processor circuit array being configured to perform matrix operations; and a control system, the control system instructing the matrix processor circuit array to process a portion of the multi-layer neural network calculation to generate a temporary data set, the control system instructing the external data interface to load a subset of tensor data from the external memory while performing the processing of the portion of the multi-layer calculation, and the control system instructing the matrix processor circuit to discard the temporary data subset not required for subsequent calculations of the multi-layer neural network calculation.

9. A neural network processor device according to claim 8, wherein the subset of tensor data from the external repository required for additional computation of the multi-layer neural network operation includes a subset of tensor input data for a new neural network operation.

10. The neural network processor device of claim 8, wherein the subset of tensor data from the external repository required for additional computation of the multi-layer neural network operation comprises a set of matrix weights for the neural network operation.

11. The neural network processor device of claim 8, wherein the control system stores a subset of the temporary data set as intermediate result tensor data in the external repository.

12. The neural network processor device according to claim 11, wherein the control system instructs the external data interface to reload the intermediate result tensor data from the external storage repository.

13. The neural network processor device of claim 8, wherein the control system instructs the matrix processor circuit to store the temporary data in the external storage repository as a final result of the multi-layer neural network operation.

14. A method for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, the method comprising the steps of: processing a first portion of multi-layer computations of a multi-layer neural network operation in a matrix processor unit to generate a temporary data set to be used in a subsequent stage of the multi-layer neural network operation; while performing said processing of said first portion of multi-layer computations, loading from said external repository a subset of tensor data required for additional computations of said multi-layer neural network operations; processing a second portion of the multi-layer computations of the multi-layer neural network operation in a matrix processor unit to generate a temporary data set to be used in a subsequent stage of the multi-layer neural network operation; while performing said processing of said second portion of multi-layer computations, storing said temporary data set to be used in a subsequent stage of said multi-layer neural network operation to said external repository; and The temporary data set is reloaded to perform the subsequent stage of the multi-layer neural network operation.

15. The method of efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation according to claim 14, wherein the subset of tensor data from the external memory repository required for additional computation of the multi-layer neural network operation includes a subset of tensor input data for a new neural network operation.

16. The method of efficiently accessing external memory of a neural network processor while executing a multi-layer neural network operation according to claim 14, wherein the subset of tensor data from the external memory repository required for additional computation of the multi-layer neural network operation includes a set of matrix weights for the neural network operation.

17. The method of efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation according to claim 14, wherein the subset of tensor data from the external memory repository required for additional computation of the multi-layer neural network operation includes a subset of data previously stored in the external memory repository.

18. The method of claim 14 for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, the method further comprising: A subset of the temporary data set is stored in the external repository as intermediate result tensor data.

19. The method of claim 18 for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, the method further comprising: Reload the intermediate result tensor data from the external repository.

20. The method of claim 14 for efficiently accessing an external memory of a neural network processor while executing a multi-layer neural network operation, the method further comprising: The temporary data is stored in the external repository as a final result of the multi-layer neural network operation.

Images