Memory for neural network processing system

Abstract

A method may include partitioning a tensor including a plurality of data values into a plurality (K) of sub-tensors, wherein each sub-tensor of the K sub-tensors may include a respective subset of the plurality of data values. The method may also include retrieving one or more first data values of the subset of data values included in a first sub-tensor of the K sub-tensors according to an access pattern associated with a neural network processor. The method may include storing the one or more first data values of the subset of data values in one of K segments of a cache memory, wherein each of the K segments may be associated with a respective one of the K sub-tensors. Further, the method may include processing, using the neural network processor, the one or more first data values of the subset of data values according to the access pattern.

Description

Technical Field

[0001] Embodiments herein relate generally to memory, and more particularly to memory for use in neural network processing systems. Background Art

[0002] In many computing systems, a neural network processor (also referred to as a "neural processing unit" ("NPU") or "neural network accelerator") retrieves data from main memory using direct memory access ("DMA") and stores the data in a local buffer from which the NPU is able to process the data. For example, the retrieved data may include a first set of weights associated with a first layer of a neural network. While processing the first set of weights, the NPU may retrieve (from the main memory) a second set of weights associated with a second layer of the neural network. However, the NPU may be idle while retrieving the second set of weights. As a result, the NPU may not be fully utilized due to the latency associated with fetching data from main memory. To reduce this latency, the NPU may pre-fetch the second set of weights from main memory before it would otherwise be needed to process the second layer of the neural network. However, pre-fetching data may present several challenges. For example, allocating space in the local buffer to store the second set of weights may be complex and may require additional local memory to store the second set of data. Further, software associated with the DMA may need to be configured to synchronize the DMA with the NPU. Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0004] An innovative aspect of the subject matter of the present disclosure can be implemented in a method. The method may include dividing a tensor including a plurality of data values ​​into a plurality (K) of sub-tensors. Each of the K sub-tensors may include a corresponding subset of the plurality of data values. The method may also include retrieving one or more first data values ​​of the subset of data values ​​included in a first sub-tensor of the K sub-tensors according to an access pattern associated with a neural network processor. The method may include storing the one or more first data values ​​of the subset of data values ​​in one of K segments of a cache memory. Each of the K segments may be associated with a corresponding one of the K sub-tensors. Further, the method may include processing the one or more first data values ​​of the subset of data values ​​using the neural network processor according to the access pattern.

[0005] Another innovative aspect of the subject matter of the present disclosure can be implemented in a system comprising a processing system and a memory. The memory may store instructions that, when executed by the processing system, cause the system to divide a tensor comprising a plurality of data values ​​into a plurality of (K) sub-tensors. Each of the K sub-tensors comprises a corresponding subset of the plurality of data values. Execution of the instructions may further cause the system to retrieve one or more first data values ​​of the subset of data values ​​included in a first sub-tensor of the K sub-tensors according to an access pattern associated with a neural network processor. Execution of the instructions may cause the system to store the one or more first data values ​​of the subset of data values ​​in one of K segments of a cache memory. Each of the K segments may be associated with a corresponding one of the K sub-tensors. Execution of the instructions may cause the system to process the one or more first data values ​​of the subset of data values ​​using the neural network processor according to the access pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The embodiments herein are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.

[0007] Figure 1 A block diagram of an example system is shown in accordance with some embodiments.

[0008] Figure 2 An example convolutional neural network (CNN) is shown that can be used to classify images according to some embodiments.

[0009] Figure 3A-3B Example operations that can be performed by a processing element (PE) are shown according to some embodiments.

[0010] Figures 4A-4D Shown is an example of how a set of PEs may be allocated according to some embodiments.

[0011] Figures 5A-5C Shows an example of how a collection of engines may be allocated according to some embodiments.

[0012] Fig. 6A An example read access pattern of a PE associated with a convolutional layer is shown according to some embodiments.

[0013] Figure 6B An example read access pattern of a PE associated with a convolutional layer is shown according to some embodiments.

[0014] Figure 6C An example write access pattern of one or more PEs associated with a convolutional layer is shown according to some embodiments.

[0015] Fig.6DAn example write access pattern of one or more PEs associated with a convolutional layer is shown according to some embodiments.

[0016] Fig. 7A Memory is shown in which example tensors are stored, according to some embodiments.

[0017] Figure 7B Memory is shown in which example tensors are stored, according to some embodiments.

[0018] Figure 8 An example block diagram of a system including a cache memory coupled to a multi-engine processor is shown in accordance with some embodiments.

[0019] Fig. 9 An example of a three-dimensional tensor is shown in accordance with some embodiments.

[0020] Fig.10 A block diagram illustrating an example cache memory according to some embodiments.

[0021] Fig.11 An example multi-engine processor is shown including a set of four engines in accordance with some embodiments.

[0022] Figures 12A-12C Memory storing mapped tensors is shown according to some embodiments.

[0023] Fig.13 Memory storing mapped tensors is shown according to some embodiments.

[0024] Fig.14 An example system is shown that includes a memory and a cache memory in accordance with some embodiments.

[0025] Fig.15 Memory is shown storing an example tensor that is virtually partitioned into multiple (K) sub-tensors according to some embodiments.

[0026] Fig.16 Memory storing example sub-tensors is shown according to some embodiments.

[0027] Fig.17 Memory storing example mapping tensors is shown according to some embodiments.

[0028] Fig.18 Example cache addresses are shown in accordance with some embodiments.

[0029] Fig.19 An example tag array is shown in accordance with some embodiments.

[0030] Fig. 20 A collection of engines subject to various restrictions when processing layers of a CNN are shown according to some embodiments.

[0031] Fig.21 A block diagram of a data processing system is shown in accordance with some embodiments.

[0032] Fig. 22 A block diagram of a data processing system is shown in accordance with some embodiments.

[0033] Fig.23 An illustrative flow diagram showing example operations 2300 is shown, in accordance with some embodiments. DETAILED DESCRIPTION

[0034] In the following description, many specific details such as examples of specific components, circuits and processes are stated to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means being directly connected to one or more intermediate components or circuits or being connected through one or more intermediate components or circuits. The terms "electronic system" and "electronic device" can be used interchangeably to refer to any system that can electronically process information. In addition, in the following description and for the purpose of illustration, specific terms are stated to provide a thorough understanding of aspects of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be required to practice example embodiments. In other examples, well-known circuits and devices are shown in block diagram form to avoid making the present disclosure obscure. Some parts of the following detailed description are provided according to the process, logic blocks, processing and other symbolic representations of the operation of the data bits in the computer memory.

[0035] These descriptions and representations are the means by which those skilled in the art of data processing most effectively convey the gist of their work to other skilled in the art. In the present disclosure, a process, logic block, progression, or the like is considered to be a self-consistent sequence of steps or instructions that produce a desired result. The steps are those that require physical manipulation of physical quantities. These quantities are typically, but not necessarily, in the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system. It should be remembered, however, that all of these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to these quantities.

[0036] Unless expressly stated otherwise, as will be apparent from the following discussion, it is to be appreciated that throughout this application, discussions utilizing terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” “derive,” and the like refer to actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms the data into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission, or display devices.

[0037] In the accompanying drawings, a single block may be described as performing one or more functions; however, in actual practice, one or more functions performed by that block may be performed in a single component or across multiple components, and / or may be performed using hardware, using software, or using a combination of hardware and software. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described below according to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention. In addition, the example input device may include components other than the components shown, including known components such as processors, memories, and the like.

[0038] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules or components may also be implemented together in an integrated logic device or individually as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium comprising instructions that, when executed, perform one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product that may include packaging material.

[0039] Non-transitory processor-readable storage media may include random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, other known storage media, and the like. The technology may additionally or alternatively be implemented at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and can be accessed, read, and / or executed by a computer or another processor.

[0040] The various illustrative logical blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein may be executed by one or more processors (or processing systems). The term "processor" as used herein may refer to any general purpose processor, special purpose processor, conventional processor, controller, microcontroller, and / or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0041] Aspects of the present disclosure provide methods and systems for operating a neural network processor using a cache memory. In some embodiments, a neural network processing system may divide a tensor including a plurality of data values ​​into a plurality of (K) sub-tensors. Each of the K sub-tensors may include a corresponding subset of the plurality of data values. In some embodiments, the plurality of data values ​​may collectively represent an input feature map ("IFM"), an output feature map ("OFM"), or a plurality of weights. The neural network processing system may also retrieve one or more first data values ​​of a subset of data values ​​included in a first sub-tensor of the K sub-tensors according to an access pattern. More specifically, in some embodiments, the neural network processing system may pre-fetch one or more first data values ​​according to the access pattern before receiving a request for the one or more first data values ​​from the neural network processor. The neural network processing system may also store the one or more first data values ​​in one of the K segments of the cache memory. Each of the K segments may be associated with a corresponding one of the K sub-tensors. Further, the neural network processor may process the one or more first data values ​​according to the access pattern.

[0042] Unlike existing neural network processing systems in which software associated with DMA must be substantially reconfigured to synchronize DMA with a neural network processor, the neural network processing system of the present disclosure requires significantly less programming effort. In addition, the neural network processing system of the present disclosure can be implemented using smaller local memory (e.g., cache memory) than the local memory (e.g., buffer) used by conventional neural network processors. Further, by pre-fetching data according to the access pattern used by the neural network processor and storing the data in the cache memory, aspects of the present disclosure can further eliminate cache miss latency.

[0043] Figure 1 A block diagram of an example system 100 is shown in accordance with some embodiments. System 100 may include a data receiver 110, a data processing system 120, and an output device 130. Data receiver 110 may include one or more sensors (e.g., an optical sensor or an acoustic sensor). Further, data receiver 110 may be configured to receive one or more signals such as input signal 101. In some embodiments, data receiver 110 may convert the received one or more signals (e.g., input signal 101) into digital input data 102 (also referred to as "input data 102"), which may represent, for example, a tensor. In some aspects, a tensor may represent an array of data values, wherein the array may include any number of dimensions and is processed by a neural network processor using a neural network.

[0044] In some embodiments, data processing system 120 may receive input data 102 from data receiver 110 and process input data 102 to generate output data 103. Figure 1 As shown in , the data processing system 120 may include a memory 121, a cache memory 122, a multi-engine processor 123, and a neural network model 124. In some embodiments, the data processing system 120 may store the input data 102 in the memory 121. The memory 121 may represent a main memory, a shared memory, or an external memory (e.g., a memory separate from or external to the data processing system 120). In some embodiments, the data processing system 120 may retrieve one or more data values ​​of the input data 102 from the memory 121 and store the retrieved one or more data values ​​in the cache memory 122. Further, in some embodiments, the data processing system 120 may retrieve one or more data values ​​from the cache memory 122 and process the one or more data values ​​retrieved from the cache memory 122 using the multi-engine processor 123 and the neural network model 124 to generate the output data 103. In some aspects, the cache memory 122, rather than the memory 121, may have sufficient bandwidth for the one or more data values ​​to be read by the multi-engine processor 123.

[0045] In some aspects, the neural network model 124 can be trained through machine learning. Machine learning (which generally includes a training phase and an inference phase) is a technique for improving the ability of a computer system or application to perform a task. During the training phase, the machine learning system is provided with one or more "answers" and a large amount of raw training data associated with the answers. The machine learning system analyzes the training data to learn a set of rules that can be used to describe each of the one or more answers. During the inference phase, the machine learning system can use the learned set of rules to infer answers from new data.

[0046] Deep learning is a specific form of machine learning in which the inference (and training) phase is performed over multiple layers, resulting in more abstract data sets in each successive layer. Deep learning architectures are often referred to as "artificial neural networks" because of the way information is processed (similar to biological nervous systems). For example, each layer of an artificial neural network may consist of one or more "neurons." The neurons may be interconnected across various layers so that input data can be processed and passed from one layer to another. More specifically, each layer of neurons may perform different transformations on the output data from the previous layer so that one or more final outputs of the neural network produce one or more desired inferences. The collection of transformations associated with the various layers of the network is referred to as a "neural network model."

[0047] A convolutional neural network ("CNN") is a type of artificial neural network that includes at least one convolutional layer and other layers, such as pooling layers and fully connected layers. A convolutional layer is a layer in which a mathematical operation called a convolution is performed. In some aspects, each layer of a CNN may be associated with one or more processing elements ("PEs"), which are computing elements configured to perform a specific operation (e.g., convolution, downsampling, or calculating probabilities or neurons).

[0048] In some embodiments, an output device 130 (e.g., a headset, a smart phone, an IoT device, or another electronic device) may receive output data 103. Output data 103 may represent an inference or prediction related to input signal 101. For example, where input signal 101 represents an image, output data 103 may represent a prediction that the image shows a particular entity (e.g., a person, an animal, or a car). In some embodiments, output device 130 may further output output data 103 using, for example, one or more speakers, a display screen, and / or other electronic devices.

[0049] Figure 2 An example CNN 200 is shown that can be used to classify images according to some embodiments. Figure 2In the example of , CNN 200 includes a convolution layer 240, convolution and pooling layers 242, 244, and 246, and a fully connected layer 248. In some aspects, the convolution layer 240 may be associated with one or more PEs configured to perform convolution. The convolution and pooling layers 242, 244, and 246 may be associated with one or more PEs configured to perform convolution and one or more PEs configured to perform downsampling. The fully connected layer 248 may be associated with one or more PEs configured to determine a classification score.

[0050] Convolutional layer 240 processes input image A to produce, for example, three output feature maps (OFMs) 241. Each of OFMs 241 is shown as a two-dimensional (2D) map having a width (or "X" dimension) and a height (or "Y" dimension). Further, each of OFMs 241 may form a corresponding channel of a three-dimensional (3D) output having a depth (or "Z" dimension), the depth being equal to the total number of OFMs 241 (or channels).

[0051] Regular and pooling layers 242 process OFM 241 to produce, for example, five new OFMs 243. Similar to OFM 241, each of OFMs 243 has a width (or X dimension) and a height (or Y dimension), and can be stacked to form a 3D output with a depth (or Z dimension) equal to the total number of OFMs 243 (or channels).

[0052] Regular and pooling layers 244 process OFM 243 to produce, for example, 10 new OFMs 245. Similar to OFMs 241 and 243, each of OFMs 245 has a width (or X dimension) and a height (or Y dimension), and can be stacked to form a 3D output with a depth (or Z dimension) equal to the total number of OFMs 245 (or channels).

[0053] Regular and pooling layers 246 process OFMs 245 to produce, for example, 20 OFMs 247. Each of OFMs 247 has a width (or X dimension), a height (or Y dimension), and can be stacked to form a 3D output with a depth (or Z dimension) equal to the total number of OFMs 247 (or channels).

[0054] The fully connected layer 248 processes the OFM 247 to generate classification scores 249. In some aspects, each of the classification scores 249 may represent the likelihood or probability that one or more features of the input image A correspond to a particular class or category (e.g., a particular type of animal). In some aspects, the CNN 200 may generate a final output based on the classification scores 249. For example, the final output may represent one or more predictions related to the input image A (e.g., a prediction that the input image A shows a lion).

[0055] like Figure 2As shown in , the output of the convolution layer 240 (e.g., OFM 241) has a large spatial dimension (e.g., X and Y dimensions) and a small number of channels relative to, for example, the output of the convolution and pooling layer 246 (e.g., OFM 247). Conversely, relative to, for example, the output of the convolution layer 240 (e.g., OFM 241), the output of the convolution and pooling layer 246 (e.g., OFM 247) has a small spatial dimension and a large number of channels.

[0056] Figure 3A and Figure 3B 2 shows example operations that can be performed by a processing element ("PE") according to some embodiments. Figure 3A and Figure 3B In the example of , each PE is illustrated as a corresponding multiplier-accumulator ("MAC") unit 350 (e.g., MAC unit 350A, 350B, or 350C). In some aspects, each of the MAC units 350A-350C represents a hardware circuit that can be configured to perform multiplication and addition operations associated with the convolutional layers of the CNN. More specifically, each of the MAC units 350A-350C multiplies a data element (e.g., data element A) by a corresponding weight. As used herein, the term "data element" may refer to a data value (e.g., a numerical value associated with a pixel or another unit). Additionally, in some embodiments described herein, the term "data element" may refer to a data value of an IFM, OFM, or weight (e.g., of a neural network model). For example, as Figure 3A As shown in FIG. 1 , MAC unit 350A multiplies data element A by weight 0 and compares the resulting product with value V retrieved from memory 360A. A1 Added to produce output O A1 . Value V A1 Represents the previous output of MAC unit 350A (such as from a previous convolution operation). Output O A1 is stored in memory 360A and can be retrieved when MAC unit 350A performs a subsequent convolution operation.

[0057] exist Figure 3A In the example of , each of the MAC units 350A-350C may perform a convolution operation based on the weights and data elements during each of a plurality of clock cycles of the computing system, which may require considerable processing power. In addition, the convolution layer may require a large amount of memory bandwidth to support the retrieval of the weights and data elements by the MAC units 350A-350C. Aspects of the present disclosure recognize that reusing data elements or weights may help save processing power and memory bandwidth.

[0058] For example, refer to Figure 3A, during a given clock cycle (e.g., clock cycle X), each of the MAC units 350A, 350B, and 350C may receive the same data element A but a different weight (e.g., weight 0, weight 1, and weight 2, respectively). In this way, data element A is "reused" between the MAC units 350A-350C. Therefore, the MAC units 350A-350C may use less processing power and memory bandwidth than if each of the MAC units 350A-350C received a different data element and a different weight during clock cycle X.

[0059] exist Figure 3B In the example of , during a given clock cycle (e.g., clock cycle Y), each of the MAC units 350A, 350B, and 350C receives the same weight (weight A) but different data elements (e.g., data element 1, data element 2, and data element 3, respectively). In this way, weight A is "reused" between the MAC units 350A-350C. Therefore, the MAC units 350A-350C can use less processing power and memory bandwidth than if each of the MAC units 350A-350C received both different weights and different data elements during clock cycle Y. In some aspects, to facilitate reuse of data elements or weights, each of the MAC units 350A-350C may be configured to process a corresponding portion of one or more OFMs or a corresponding set of OFMs.

[0060] Figures 4A-4D An example of how a set of PEs may be allocated according to some embodiments is shown. More specifically, Figures 4A-4D An example is shown of how each PE of a set of PEs may be allocated within a layer (e.g., a convolutional layer) of a CNN to produce a corresponding portion of one or more OFMs or a corresponding set of OFMs. In some aspects, each of the PEs may be Figure 3A and Figure 3B An example of any MAC unit in MAC units 350A-350C of the CNN. Further, each of the PEs may be synchronized to perform the same operation (e.g., a convolution operation) during a given clock cycle. Aspects of the present disclosure recognize that the operation performed by each PE of the CNN is predictable or known (or determined) before processing a layer of the CNN.

[0061] Figure 4A An example 400A shows how each PE of a set of PEs can be allocated to produce a corresponding set of OFMs. This allocation may be referred to as "OFM allocation" or "channel allocation." Figure 4A The example shows eight OFM allocations (OFM 0-OFM7) for four PEs (PE0-PE3). However, in actual implementations, Figure 4AThe example OFM allocation technique can be applied to any number of PEs. Figure 4A As shown in , PE0 is based on the input (for example, reference Figure 3A and Figure 3B Similarly, PE1 generates each of OFM2 and OFM 3 based on the input, PE2 generates each of OFM4 and OFM 5 based on the input, and PE3 generates each of OFM 6 and 7 based on the input. In some aspects, each of PE0-PE3 can use different weights to process the same data element during a given clock cycle (such as with reference to FIG. Figure 3A Therefore, Figure 4A The example OFM allocation of may be well suited for generating a network with a large number of channels (e.g., Figure 2 The output of any layer of a CNN of OFM 247).

[0062] Figure 4B An example 400B shows how each PE of a set of PEs can be allocated to produce a respective portion of the spatial dimension of each of OFMs 0-3. This allocation may be referred to as "spatial allocation." Figure 4B The example shows four space allocations (parts A, B, C, and D) for each of the four PEs (PE0-PE3) and OFMs 0-3, respectively. However, in actual implementations, Figure 4B The example space allocation technique can be applied to any number of PEs. Figure 4B As shown in , PE0 is based on the input (e.g., as shown in Figure 3A and Figure 3B 3) to generate portion A of each of OFM0-3 based on the inputs, PE1 generates portion B of each of OFM 0-3 based on the inputs, PE2 generates portion C of each of OFM 0-3 based on the inputs, and PE3 generates portion D of each of OFM 0-3 based on the inputs. In some aspects, each of PE0-PE3 may use the same weights to process different data elements during a given clock cycle (such as with reference to FIG. 1 ). Figure 3B Therefore, Figure 4B The example spatial allocation of may be quite suitable for generating channels with large spatial dimensions and a small number of channels (e.g., Figure 2 The output layer of the CNN of OFM 241).

[0063] Figure 4CExample 400C shows how each consecutive PE of a set of PEs can be allocated during a given time period to produce corresponding data elements along the X dimension of the OFM, where each corresponding data element is placed adjacent to each other. This allocation may be referred to as "stride allocation". Figure 4C The example shows four PEs (PE0-PE3) and 64 spatial allocations for each of OFM0 and OFM1 (at the intersections of X=0 to X=7 and Y=0 to Y=7). However, in actual implementations, Figure 4B The example space allocation technique can be performed by any number of PEs. Figure 4C As shown in FIG. , during a first time period (eg, a plurality of clock cycles) and for OFM0 and Y=0, PE0 generates a clock signal based on an input (eg, as shown in FIG. 1 ). Figure 3A and Figure 3B , Z=0. Similarly, during a first time period and for OFM 0 and Y=0, PE1 generates a data element at X=1, Z=0 based on input; PE2 generates a data element at X=2, Z=0 based on input; and PE3 generates a data element at X=3, Z=0 based on input. Further, during a second time period (e.g., a plurality of clock cycles) and for OFM 0 and Y=0, PE0 generates a data element at X=4, Z=0 based on input; PE1 generates a data element at X=5, Z=0 based on input; PE2 generates a data element at X=6, Z=0 based on input; and PE3 generates a data element at X=7, Z=0 based on input. During subsequent time periods, each of PE0-PE3 repeats this process until each data element of OFM 0 and 1 is generated for Y=0 to Y=7.

[0064] In some aspects, each of PE0-PE3 may use the same weights to process different data elements (such as reference 1) during a given clock cycle. Figure 3B). Further, in the case where the weights have dimensions greater than 1×1 (e.g., one unit wide and one unit high), stride allocation can facilitate data reuse in subsequent clock cycles. For example, a first data element can be processed by PE3 at (X=3, Y=0, Z=0) during a first clock cycle, by PE2 at (X=2, Y=0, Z=0) during a second clock cycle, by PE1 at (X=1, Y=0, Z=0) during a third clock cycle, and by PE0 at (X=0, Y=0, Z=0) during a fourth clock cycle. Similarly, the second data element processed by PE3 at (X=3, Y=0, Z=0) during the second clock cycle may also be processed by PE2 at (X=2, Y=0, Z=0) during the third clock cycle, by PE1 at (X=1, Y=0, Z=0) during the fourth clock cycle, and by PE0 at (X=0, Y=0, Z=0) during the fifth clock cycle, etc. In some aspects, a multi-engine processor (e.g., Figure 1 The multi-engine processor 123 may include special hardware (e.g., connections between PE0-PE3) to facilitate such data reuse with stride allocation. In some aspects, the special hardware may define the number of PEs (or stride size) for a given engine of the multi-engine processor. Further, in some embodiments, the special hardware may not be configured to dynamically change when used to process a layer of a CNN.

[0065] Figure 4D An example 400D shows how a set of PEs can be allocated to generate a portion of the space dimension of each of OFMs 0-3. This allocation may be referred to as a "combined allocation" because the allocation is similar to the combination of OFM allocation, space allocation, and stride allocation discussed above. Figure 4D The example shows eight PEs (PE0-PE7) and four space allocations (parts A, B, C, and D) for each of OFM0-3. However, in actual implementations, Figure 4D The example combined allocation technique can be applied to any number of PEs. Figure 4D As shown in , PE0 is based on the input (e.g., as shown in Figure 3A and Figure 3B 1) to generate part A of each OFM 0-1 based on the input, PE2 generates part C of each OFM 0-1 based on the input, and PE3 generates part D of each OFM 0-1 based on the input. Further, PE4 generates part A of each OFM 2-3 based on the input, PE5 generates part B of each OFM 2-3 based on the input, and PE6 ( Figure 4D) generates a portion C of each of OFM2-3 based on the input, and PE7 generates a portion D of each of OFM2-3 based on the input. In some aspects, Figure 4D The example combination allocation of may be quite suitable for generating channels with medium spatial dimensions and medium number of channels (e.g., Figure 2 The output layer of the CNN of OFM 243 or OFM 245).

[0066] In some aspects, PE allocation (e.g., OFM allocation, space allocation, stride allocation, or a combination of allocations) can be designed to reduce memory bandwidth and maintain high PE utilization (such that few or no PEs are idle while a layer is being processed). Further, in some aspects, each of a plurality of different groups of PEs can be allocated to process a respective portion of one or more OFMs or a respective set of OFMs. Each group of PEs can include two or more PEs collectively referred to as an "engine."

[0067] Figures 5A-5C shows an example of how a collection of engines may be allocated according to some embodiments. More specifically, Figures 5A-5C An example is shown of how each engine of a set of engines may be allocated within a layer (e.g., a convolutional layer) of a CNN to generate a corresponding portion of one or more OFMs or a corresponding set of OFMs. As described above, each engine in the set of engines may include a corresponding group of two or more PEs, where each of the two or more PEs may be Figure 3A and Figure 3B An example of any MAC unit in MAC units 350A-350C of the CNN. Further, each of the PEs of each engine can be synchronized to perform the same operation (e.g., a convolution operation) during a given clock cycle. Aspects of the present disclosure recognize that the operations performed by each engine of the CNN are predictable or known (or determined) before processing the layers of the CNN.

[0068] Figure 5A An example 500A shows how each engine of a set of engines can be allocated to produce a corresponding set of OFMs. This allocation may be referred to as an "OFM allocation" or "channel allocation", as in reference to Figure 4A The distribution described is similar. Figure 5A The example shows eight OFM allocations (OFM0-OFM7) for four engines (ENG0-ENG3). However, in actual implementation, Figure 5A The example OFM allocation technique can be applied to any number of engines. Figure 5A As shown in FIG, ENG0 is based on input (e.g., reference Figure 3A and Figure 3BSimilarly, ENG1 generates each of OFM 2 and OFM 3 based on the input, ENG2 generates each of OFM 4 and OFM 5 based on the input, and ENG3 generates each of OFM 6 and OFM 7 based on the input. In some aspects, each PE of ENG0-ENG3 can use different weights to process the same data element during a given clock cycle (such as with reference to FIG. Figure 3A Therefore, Figure 5A The example OFM allocation of may be well suited for generating a network with a large number of channels (e.g., Figure 2 The output of any layer of a CNN of OFM 247).

[0069] Figure 5B Example 500B shows how each engine of a set of engines can be allocated to generate a corresponding portion of the spatial dimension of each of OFMs 0-3. This allocation may be referred to as a "spatial allocation," as in reference to Figure 4B The distribution described is similar. Figure 5B The example shows four space allocations (parts A, B, C, and D) for four engines (ENG0-ENG3) and each of OFM 0-3, respectively. However, in actual implementations, Figure 5B The example space allocation technique can be applied to any number of engines. Figure 5B As shown in FIG, ENG0 is based on input (e.g., as shown in FIG. Figure 3A and Figure 3B Similarly, ENG1 generates portion B of each of OFM 0-3 based on the input, ENG2 generates portion C of each of OFM 0-3 based on the input, and ENG3 generates portion D of each of OFM 0-3 based on the input. In some aspects, each PE of ENG0-ENG3 may use the same weights to process different data elements during a given clock cycle (such as reference 1). Figure 3B Therefore, Figure 5B The example spatial allocation of may be quite suitable for generating channels with large spatial dimensions and a small number of channels (e.g., Figure 2 The output layer of the CNN of OFM 241).

[0070] Figure 5C Example 500C shows how a collection of engines can be allocated to generate a portion of the space dimension of each of OFMs 0-3. This allocation may be referred to as a "combined allocation," as in reference to Figure 4D The distribution described is similar. Figure 5CThe example shows eight engines (ENG0-ENG7) and four space allocations (parts A, B, C, and D) for each of OFMs 0-3. However, in actual implementations, Figure 5C The example combined allocation technique can be applied to any number of engines. Figure 5C As shown in FIG, ENG0 is based on input (e.g., as shown in FIG. Figure 3A and Figure 3B 1) to generate part A of each of OFM 0-1 based on the input, ENG2 generates part C of each of OFM 0-1 based on the input, and ENG3 generates part D of each of OFM 0-1 based on the input. Further, ENG4 generates part A of each of OFM 2-3 based on the input, ENG5 generates part B of each of OFM 2-3 based on the input, and ENG6 ( Figure 5C ) generates a portion C of each of OFM 2-3 based on the input, and ENG7 generates a portion D of each of OFM 2-3 based on the input. In some aspects, Figure 5C The example combination allocation of the engine may be quite suitable for generating channels with medium spatial dimensions and medium number of channels (e.g., Figure 2 The output layer of the CNN of OFM 243 or OFM 245).

[0071] In some aspects, engine allocation (e.g., OFM allocation, spatial allocation, stride allocation, or a combination of allocations) can be designed to reduce memory bandwidth and maintain high PE utilization (such that when a layer is being processed, few or no PEs of the engine are idle). Further, in some aspects, each of one or more engines of a set of engines can dynamically change its PE allocation. For example, each engine of a set of engines can be assigned a spatial allocation for the first layer of a CNN, and an OFM allocation for subsequent layers of the CNN (e.g., the third layer, the fourth layer, etc.).

[0072] In some aspects, the access mode of a CNN refers to the order or sequence in which one or more engines of a neural network processor read (or receive) and write (or generate) data elements of a layer (e.g., a convolutional layer) of the CNN. Thus, the access mode of a layer of a CNN may include both a read access mode and a write access mode. As used herein, the term "access mode of a CNN" may also be referred to as "CNN access mode." In some aspects, a CNN access mode may be determined based on the PE allocation assigned to one or more engines, as described above with reference to Figures 4A-5CAs discussed above. The CNN access mode may also be determined based on the dimensions of the kernel (or filter or weight) sub-tensor associated with the convolutional layer. For example, the CNN access mode may be determined based on one or more dimensions (e.g., width, height, depth, or index (or reference) number) of the kernel sub-tensor associated with the convolutional layer.

[0073] Fig. 6A PEs associated with a convolutional layer (e.g., PE A ) of an example read access pattern 600A. More specifically, the read access pattern 600A represents a PE A Can read kernel X A The weight of IFMY A A subset of the data elements of and produces OFMZ A In some embodiments, kernel X A 、IFMY A and OFM Z A Each of can represent a corresponding two-dimensional sub-tensor. However, in actual implementation, the kernel X A 、IFMY A and OFMZ A Each of can have any number of dimensions.

[0074] exist Fig. 6A In the example, kernel X A is shown as an array of weights, including 3 rows of weights (representing kernel X A X dimension) and 3 columns of weights (representing kernel X A The Y dimension of the kernel is 2, which is 9 weights in total. Each of the 9 weights corresponds to a corresponding coordinate pair (e.g., (X coordinate, Y coordinate)). For example, the weight corresponding to the position (2, 0) represents the position at the kernel X. A The weight at (X=2, Y=0) within.

[0075] IFM A is shown as an array of data elements, including 5 rows of IFM data elements (representing IFM Y A X dimension) and 5 columns of IFM data elements (representing IFM Y A ), for a total of 25 IFM data elements. A Each of the 25 IFM data elements corresponds to a corresponding coordinate pair. For example, the IFM data element corresponding to position (4, 0) represents the coordinates located at IFMY. A The IFM data element at (X=4, Y=0) within.

[0076] like Fig. 6A As shown in A, which is shown as an array of data elements, including 3 rows of OFM data elements (representing OFM Z A X dimension) and 3 columns of OFM data elements (representing OFM Z A Y dimension), a total of 9 OFM data elements. A Similar to the data elements of OFM, each of the 9 OFM data elements corresponds to a corresponding coordinate pair. For example, the OFM data element corresponding to the position (0, 0) represents the coordinate position at OFM Z. A OFM data element at (X=0, Y=0) within.

[0077] In some embodiments, PE A Can be based on the read access mode 600A based on the core X A The weight of IFMY A A subset of the IFM data elements is used to generate the OFMZ A The read access pattern 600A represents the OFM data element at (0,0) of PE A First, along the kernel X A and IFM Y A Each of the X dimensions and then along the kernel X A and IFM Y A An access pattern for reading weights and IFM data elements in each Y dimension (also known as an "XY read access pattern").

[0078] According to the read access mode 600A, during the first clock cycle, PE A Can read kernel X A The weight at position (0, 0) and IFMY A During the second clock cycle, PE A Can read kernel X A The weight at position (1, 0) and IFMY A The IFM data element at position (1, 0) of PE A Can read kernel X A The weight at position (2, 0) and IFMY A Therefore, during the first, second, and third clock cycles, PE A Along the kernel X A The weights are read from the X dimension (for Y=0) and along the IFM Y A The IFM data elements are read from the X dimension (for Y=0) of .

[0079] Next, PE AThe read access mode 600A may be followed along the core X A and IFM Y A In other words, during the fourth clock cycle, PE A Can read kernel X A The weight at position (0, 1) and IFM Y A During the fifth clock cycle, PE A Can read kernel X A The weight at position (1, 1) and IFM Y A The IFM data element at position (1, 1) of PE A Can read kernel X A The weight at position (2, 1) and IFM Y A The IFM data element at position (2, 1).

[0080] Next, PE A Core X can be read in read access mode 600A A The residual weight and IFM Y A That is, during the seventh clock cycle, PE A Can read kernel X A The weight at position (0, 2) and IFMY A During the eighth clock cycle, PE A Can read kernel X A The weight at position (1, 2) and IFMY A The IFM data element at position (1, 2) of ; and during the ninth clock cycle, PE A Can read kernel X A The weight at position (2, 2) and IFMY A In some embodiments, PE A Or one or more other PEs may be used to generate OFMZ based on, for example, the read access mode 600A and the write access mode (eg, A The order of OFM data elements in IFMY A Slide kernel X on top A , generating OFM Z A The remaining OFM data elements.

[0081] Figure 6B PEs associated with a convolutional layer (e.g., PE B) of the example read access pattern 600B. More specifically, the read access pattern 600B represents the PE B Can read kernel X B The weight of IFMY B A subset of the data elements of and produces OFMZ B The sequence of one or more data elements of a PE B , Kernel X B 、IFMY B and OFMZ B They can be Fig. 6A PE A , Kernel X A 、IFMY A OFM Z A Embodiment of the invention.

[0082] In some embodiments, PE B Can be based on the read access mode 600B based on the core X B The weight of IFMY B A subset of the IFM data elements is used to generate the OFMZ B The OFM data element at (0, 0) of the read access mode 600B represents the PE B First, along the kernel X B and IFMY B Each of the Y dimensions and then along the kernel X B and IFM Y B An access pattern for reading weights and IFM data elements separately for each of the X dimensions in (also referred to as a "YX read access pattern").

[0083] According to the read access mode 600B, during the first clock cycle, PE B Can read kernel X B The weight at position (0, 0) and IFMY B During the second clock cycle, PE B Can read kernel X B The weight at position (0, 1) and IFMY B The IFM data element at position (0, 1) of ; and during the third clock cycle, PE B Can read kernel X B The weight at position (0, 2) and IFMY B Therefore, during the first, second, and third clock cycles, PE B Along the kernel X B The weight is read from the Y dimension (for X=0) and along the IFM Y BThe IFM data elements are read from the Y dimension (for X=0).

[0084] Next, PE B The read access mode 600B may be followed along the core X B and IFMY B In other words, during the fourth clock cycle, PE B Can read kernel X B The weight at position (1, 0) and IFM Y B During the fifth clock cycle, PE B Can read kernel X B The weight at position (1, 1) and IFMY B The IFM data element at position (1, 1) of PE B Can read kernel X B The weight at position (1, 2) and IFM Y B The IFM data element at position (1, 2).

[0085] Next, PE B Core X can be read in read access mode 600B B The residual weight and IFM Y B That is, during the seventh clock cycle, PE B Can read kernel X B The weight at position (2, 0) and IFMY B During the eighth clock cycle, PE B Can read kernel X B The weight at position (2, 1) and IFMY B The IFM data element at position (2, 1) of B Can read kernel X B The weight at position (2, 2) and IFMY B In some embodiments, PE B Or one or more other PEs may be used to access the IFM Y by, for example, reading the access mode 600B and writing the access mode 600B. B Sliding Kernel X B , generating OFM Z B The remaining OFM data elements.

[0086] Figure 6C600C is an example write access pattern of one or more PEs associated with a convolutional layer according to some embodiments. More specifically, the write access pattern 600C indicates that one or more PEs (e.g., of one or more engines) may generate OFMZ C In some embodiments, one or more PEs may be Fig. 6A and Figure 6B PE A and / or PE B Embodiment of OFMZ C They can be Fig. 6A and Figure 6B OFM Z A and / or OFM Z B Further, in some embodiments, OFM Z C Can represent three-dimensional subtensors.

[0087] exist Figure 6C In the example, OFMZ C is shown as an array of data elements, including 3 rows of data elements (representing OFM Z C X dimension), 3 columns of OFM data elements (indicating OFM Z C Y dimension) and 3 channels of OFM data elements (indicating OFM Z C Z dimension), a total of 27 OFM data elements. Figure 6B OFM Z B The data elements of OFM Z are similar. C Each of the OFM data elements corresponds to a corresponding coordinate triple. For example, the OFM data element corresponding to the position (0, 0, 0) represents a coordinate triple located at OFMZ. C OFM data element at (X=0, Y=0, Z=0) within.

[0088] In some embodiments, one or more PEs may generate OFMZ based on the weights and IFM data elements. C OFM data elements (such as reference Fig. 6A and Figure 6B The write access mode 600C represents one or more PEs first accessing the OFMZ C Z dimension, and then along OFM Z C Y dimension and then along OFM Z C An access pattern for writing (or generating) OFM data elements using the X dimension of the OFM (also known as the "ZYX write access pattern").

[0089] According to the write access mode 600C, one or more PEs may first (sequentially or simultaneously) generate OFM ZC Therefore, one or more PEs first read the OFM data elements at positions (0, 0, 0), (0, 0, 1), and (0, 0, 2) along the OFM Z. C One or more PEs may then generate OFM data elements along the Z dimension of the OFM (for X=0, Y=0). C The Y dimension of the OFM may generate OFM data elements at positions (0, 1, 0), (0, 1, 1), and (0, 1, 2) and then at positions (0, 2, 0), (0, 2, 1), and (0, 2, 2) (sequentially or simultaneously). Subsequently, one or more PEs may generate OFM data elements along the Y dimension of the OFM Z. C The X dimension of the OFM data element is generated at positions (1,0,0), (1,0,1) and (1,0,2) and then at positions (1,1,0), (1,1,1) and (1,1,2) (sequentially or simultaneously). In some embodiments, one or more PEs may generate OFMZ according to the write access mode 600C. C The remaining OFM data elements.

[0090] Fig.6D 600D is an example write access pattern of one or more PEs associated with a convolutional layer according to some embodiments. More specifically, the write access pattern 600D indicates that one or more PEs (eg, of one or more engines) may generate OFMZ D In some embodiments, one or more PEs may be Fig. 6A and Figure 6B PE A and / or PE B Embodiment of OFMZ D They can be Fig. 6A and Figure 6B OFM Z A and / or OFM Z B In some embodiments, OFM Z D Can represent three-dimensional subtensors.

[0091] In some embodiments, one or more PEs may generate OFMZ based on the weights and IFM data elements. D OFM data elements (such as reference Fig. 6A and Figure 6B Write access pattern 600D represents an access pattern in which one or more PEs first write (or generate) OFM data elements along the X dimension of OFM ZD, then along the Z dimension of OFM ZD, and then along the Y dimension of OFM ZD (also referred to as "XZY write access pattern").

[0092] According to the write access mode 600D, one or more PEs may first (sequentially or simultaneously) generate OFM Z D Therefore, one or more PEs first read the OFM data elements at positions (0, 0, 0), (1, 0, 0), and (2, 0, 0) along the OFM Z. D One or more PEs may then generate OFM data elements along the X dimension of the OFM Z (for Y=0, Z=0). D The Z dimension of the OFM generates OFM data elements at positions (0,0,1), (1,0,1), and (2,0,1) and then at positions (0,0,2), (1,0,2), and (2,0,2) (sequentially or simultaneously). Subsequently, one or more PEs may generate OFM data elements along the Z dimension of the OFM. D The Y dimension of the OFM data elements is generated at positions (0, 1, 0), (1, 1, 0) and (2, 1, 0) and at positions (0, 1, 1), (1, 1, 1) and (2, 1, 1) (sequentially or simultaneously). In some embodiments, one or more PEs may generate OFM Z according to the write access mode 600D. D The remaining OFM data elements.

[0093] Fig. 7A 14 shows a memory 721A in which an example tensor 702A is stored, according to some embodiments. The memory 1404 may be, for example, a main memory, a shared memory, or an external memory. In some embodiments, the memory 721A may be Figure 1 An example of the memory 121.

[0094] Tensor 702A is shown as a two-dimensional array having a width and a height (representing the X and Y dimensions, respectively). Fig. 7A In the example of , tensor 702A is depicted as 8 data elements wide and 4 data elements high. However, in actual implementations, tensor 702A may have any dimensions. In some embodiments, where tensor 702A includes one or more weights (or kernels), tensor 702A may have a width, height, depth, and / or index (or reference) number. In some embodiments, memory 721A may store multiple tensors.

[0095] like Fig. 7A As shown in FIG. 1 , each data element is labeled with a corresponding number ranging from 1 to 32. In some embodiments, each of the numbers 1-32 may represent a tag of a tag array. In some embodiments, each of the 32 data elements may be an example of a data element of an IFM, OFM, or kernel (as described above). Further, in some embodiments, each of the 32 data elements may be labeled with a number different from Fig. 7A The arrangement (or configuration) shown in is stored in the memory 721A.

[0096] In some embodiments, the tensor 702A may be virtually divided (or sharded) into multiple (K) sub-tensors. Fig. 7A As shown in FIG. , the tensor 702A can be virtually divided into K=2 sub-tensors (eg, sub-tensors 702A ST0 and Subtensor 702A ST1 ). Subtensor 702A ST0 and 702A ST1 Each of has the same width and height dimensions. In some embodiments, the number of sub-tensors (K) associated with tensor 702A may be preconfigured. For example, the number of sub-tensors (e.g., K=2) may be preconfigured to be equal to the number of associated multi-engine processors (e.g., Figure 1 The number of engines in the multi-engine processor 123 (e.g., 2 engines). Further, the dimension of each of the K sub-tensors may be based on the CNN access mode and the multi-engine processor associated with the memory 721A (e.g., Figure 1 The allocation of engines used in the multi-engine processor 122 (e.g., OFM allocation, spatial allocation, stride allocation, or combined allocation) is predetermined.

[0097] Figure 7B 7 shows a memory 721B in which an example tensor 702B is stored, according to some embodiments. The memory 721B and the tensor 702B may be Fig. 7A An example of a memory 721A and a tensor 702A. Figure 7B As shown in FIG. 702B, the tensor 702B is divided into the following eight sub-tensors: 702B ST0 、702B ST1 、702B ST2 、702B ST3 、702B ST4 、702B ST5 、702B ST6 (not shown) and 702B ST7 In some embodiments, subtensor 702B ST0 -702B ST7 Each of may be retrieved from memory 721B and stored in a cache memory (e.g., Figure 1 In the corresponding segment of the cache memory 123).

[0098] Figure 8 An example block diagram of a system 800 according to some embodiments is shown, the system including a cache memory 822 coupled to a multi-engine processor 823. The cache memory 822 and the multi-engine processor 823 may be Figure 1 122 and a multi-engine processor 123. Figure 8As shown in , cache memory 822 may be communicatively coupled to multi-engine processor 823 via one or more multiplexers 825 (also referred to as “mux 825 ”).

[0099] In some embodiments, cache memory 822 may represent L1 cache memory. Further, cache memory 822 may be configured to store only one or more data elements of a particular tensor at a given time. For example, cache memory 822 may be configured to store Figure 7B 702B (e.g., data elements or weights of an IFM or OFM). Figure 8 Only one cache memory 822 is shown coupled to the multi-engine processor 823 via mux 825, but in some other embodiments, multiple cache memories 822 may be coupled to the multi-engine processor 823 via mux 825, wherein each of the cache memories 822 stores one or more data elements of a corresponding tensor.

[0100] In some embodiments, cache memory 822 may include or be divided into multiple (K) segments (or regions) of memory, wherein each of the K segments is configured to store one or more data elements of a corresponding sub-tensor. Figure 8 As shown in , cache memory 822 includes eight segments (segments 0-7) of cache memory. In some embodiments, segment 0 of cache memory 822 may include a sub-tensor (e.g., Figure 7B Sub-tensor 702B ST0 ) of one or more data elements (e.g., S0_0, ..., S0_7, ...); and segment 1 of cache memory 822 may include a sub-tensor (e.g., Figure 7B Sub-tensor 702B ST1 ) of one or more data elements (e.g., S1_0, ..., S1_7, ...). Segment 2 of cache memory 822 may include a sub-tensor (e.g., Figure 7B Sub-tensor 702B ST2 ) of one or more data elements (e.g., S2_0, ..., S2_7, ...); and segment 3 of cache memory 822 may include a sub-tensor (e.g., Figure 7B Sub-tensor 702B ST3 ) of one or more data elements (e.g., S3_0, ..., S3_7, ...). Continuing with the example, segment 4 of cache memory 822 may include a sub-tensor (e.g., Figure 7B Sub-tensor 702B ST4 ) of one or more data elements (e.g., S4_0, ..., S4_7, ...); and segment 5 of cache memory 822 may include a sub-tensor (e.g., Figure 7B Sub-tensor 702BST5 ) of one or more data elements (e.g., S5_0, ..., S5_7, ...). Further, segment 6 of cache memory 822 may include sub-tensors (e.g., Figure 7B Sub-tensor 702B ST6 ) of one or more data elements (e.g., S6_0, ..., S6_7, ...); and segment 7 of cache memory 822 may include a sub-tensor (e.g., Figure 7B Sub-tensor 702B ST7 ) of one or more data elements (e.g., S7_0, ..., S7_7, ...). In some embodiments, each of the one or more data elements stored in each of segments 0-7 may be accessed from a memory (e.g., Figure 7B is taken out from memory 721B).

[0101] In some embodiments, the multi-engine processor 823 may include a plurality (K) of engines, where the number (K) also corresponds to the number of segments included in the cache memory 822 and the number of sub-tensors into which a tensor may be partitioned in memory (as described above with reference to Fig. 7A and Figure 7B as described). Figure 8 As shown in , the multi-engine processor 823 includes K=8 engines (e.g., engines 0-7). In some embodiments, each of the engines 0-7 may be configured to read from and write to the cache memory 822. For example, in some embodiments, each of the engines 0-7 may be configured to read from the same segment (e.g., segment 0) of the cache memory 822 concurrently without contention, and / or each of the engines 0-7 may be configured to read from different segments of the cache memory 822 concurrently without contention. Further, in some embodiments, when each of the engines 0-7 of the multi-engine processor 823 completes reading one or more data elements of a particular segment of the cache memory 822, each of the engines 0-7 may proceed to read data from a different segment of the cache memory 822 without contention.

[0102] Further, in some embodiments, each of the engines 0-7 of the multi-engine processor 823 may be configured to write to a corresponding segment of the cache memory 822 without contention. For example, in the case where each of the engines 0-7 of the multi-engine processor 823 generates a corresponding sub-tensor of the OFM, each of the engines 0-7 may store the corresponding sub-tensor in a different segment of the cache memory 822.

[0103] As mentioned above Figure 7B As discussed, for a given layer of a CNN, the dimensions of the K sub-tensors into which the tensor is divided may be based on a multi-engine processor (e.g., Figure 8The allocation (e.g., channel allocation, spatial allocation, stride allocation, or combined allocation) of the engines in the multi-engine processor 823 of the multi-engine processor 823 is pre-configured. In other words, for a particular layer of the CNN, the dimensions of the K sub-tensors into which the tensor is divided can be pre-configured to support (or enable) engine allocation, and in turn support (or enable) the access mode used by each of the engines in the multi-engine processor 823.

[0104] See for example Fig. 9 The three-dimensional tensor 902A including the data elements of the IFM (also referred to as “IFM tensor 902A”) may be divided into eight sub-tensors (e.g., 902 ST0 -902 ST7 More specifically, the IFM tensor 902A may be allocated in memory based on the combination (e.g., Figure 7B The memory 721B) is divided into sub-tensors 902 ST0 -902 ST7 . Further, eight sub-tensors (e.g., 902 ST0 -902 ST7 ) may include a corresponding set of data elements of IFM tensor 902A. Further, during a certain time period, eight sub-tensors (e.g., 902 ST0 -902 ST7 ) may be stored in a corresponding segment (e.g., segments 0-7) of the cache memory, such as Fig. 9 This partitioning can facilitate the use of a combination of allocations to allocate a set of eight engines (eg, ENG0-ENG7) of a multi-engine processor, such as Fig. 9 That is, each of the engines ENG0-ENG7 can be based on the sub-tensor 902 without contention. ST0 -902 ST7 The data elements of are used to generate the corresponding sub-tensor (or part) of OFM tensor 902B.

[0105] Fig.10 A block diagram showing an example cache memory 1022 according to some embodiments. The cache memory 1022 may be Figure 1 and Figure 8 In some embodiments, the cache memory 1022 may include or be virtually divided into K segments of memory, as described above with reference to Figure 8 More specifically, Fig.10 As shown in , cache memory 1022 may include K=4 segments (e.g., segments 0-3), wherein each of segments 0-3 may store a corresponding sub-tensor (e.g., Figure 7B Sub-tensor 702B ST0). For example, segment 0 may store data elements S0_0, ..., S0_7, ... of the first sub-tensor, and segment 1 may store data elements S1_0, ..., S1_7, ... of the second sub-tensor. Further, segment 2 may store data elements S2_0, ..., S2_7, ... of the third sub-tensor, and segment 3 may store data elements S3_0, ..., S3_7, ... of the fourth sub-tensor. However, in some embodiments, the size (or storage capacity) of each of segments 0-3 may be smaller than the size of the first to fourth sub-tensors (or the amount of data in the first to fourth sub-tensors), respectively. Further, in some embodiments, each data element (e.g., data S0_0 or data S3_7) stored in the cache memory 1022 may represent one or more bytes (or words) of data.

[0106] like Fig.10 As shown in , cache memory 1022 is not only divided into segments 0-3, but also divided into at least cache lines 1-4. In some embodiments, each of cache lines 1-4 may include multiple (N) data elements from each of segments 0-3. More specifically, as Fig.10 As shown in , each of cache lines 1-4 includes N=2 data elements from each of segments 0-3. For example, cache line 1 includes data (i) S0_0 and (ii) S0_1 for segment 0; data (i) S1_0 and (ii) S1_1 for segment 1; data (i) S2_0 and (ii) S2_1 for segment 2; and data (i) S3_0 and (ii) S3_1 for segment 3. In some embodiments, N may be equal to an integer greater than or equal to 1 (e.g., N=1, 2, 3, ...). Further, in some embodiments, N may be equal to a power of 2.

[0107] Fig.10 As further shown, each data element (e.g., S0_0) of cache memory 1022 may be associated with a stride width. In some embodiments, the stride width may represent the number of bytes (or words) within a data element (e.g., S0_0) that an engine of a multi-engine processor can read during one clock cycle (also referred to as the "data element size"). In other words, the stride width may represent the product of the number of PEs in an engine (also referred to as the "stride size") and the data element size (e.g., stride width=stride size*data element size).

[0108] Fig.10 It is also shown that for each segment of cache memory 1022, the N data elements of each segment of a given cache line (e.g., cache line 1) (e.g., N=2 data elements of segment 0) form a segment. For simplicity, Fig.10Only two segments are annotated in the figure. In some embodiments, each segment may represent (or determine) the spatial locality of the cache memory 1022. For example, when the PE of the engine reads the data element data S0_0 of a specific segment of segment 0, both data S0_0 and data S0_1 can be retrieved for the PE. In other words, the data element requested by the PE (in a specific segment of the segment) and all other data elements in the segment can be supplied to the PE based on the principle of spatial locality. In some embodiments, the segment size may represent the product of N and the stride width (for example, segment size = N*stride width). Although Fig.10 As shown, the cache memory 1022 includes K=4 segments and N=2 data elements per segment, but in actual implementations, the cache memory 1022 may include any number of segments and any number of data elements per segment.

[0109] Fig.11 An example multi-engine processor 1123 is shown including a set of four engines (eg, ENG0-ENG3) according to some embodiments. The multi-engine processors 1123 may be Figure 1 and Figure 8 Embodiments of the multi-engine processor 123 and / or 823. In some aspects, the four engines ENG0-ENG3 may be allocated according to the OFM allocation, such that ENG0 generates each of OFM 0 and OFM 1 based on input, ENG1 generates each of OFM 2 and OFM 3 based on input, ENG2 generates each of OFM 4 and OFM 5 based on input, and ENG3 generates each of OFM 6 and OFM 7 based on input.

[0110] In some embodiments, each of ENG0-ENG3 may use the same coordinate system. Fig.11 As shown in FIG. 1 , each of ENG0-ENG3 may use a corresponding three-dimensional coordinate system consisting of X, Y, and Z axes (also referred to as an "X, Y, Z coordinate system"). In some embodiments, the origin of each of the X, Y, and Z coordinate systems (e.g., where X=0, Y=0, and Z=0) may coincide with an OFM data element located in the front, upper left corner of each OFM (e.g., OFM 0, 2, 4, and 6), as shown in FIG. Fig.11 Although Fig.11 The origin of each of the X, Y, and Z coordinate systems in is represented by a rectangle, but in some embodiments, the origin of each of the X, Y, and Z coordinate systems (or other locations) may be represented by a different shape.

[0111] Further, when engines ENG0-ENG3 generate OFM 0-7, each of engines ENG0-ENG3 may operate in lock step. That is, when ENG0 generates a data element of OFM0 located at (0, 0, 0) in the X, Y, Z coordinate system of ENG0, ENG1 may simultaneously generate a data element of OFM2 located at (0, 0, 0) in the X, Y, Z coordinate system of ENG1; ENG2 may simultaneously generate a data element of OFM 4 located at (0, 0, 0) in the X, Y, Z coordinate system of ENG2; and ENG3 may simultaneously generate a data element of OFM 6 located at (0, 0, 0) in the X, Y, Z coordinate system of ENG 3. In other words, during any given time period, ENG0-ENG3 may generate data elements of OFM0, 2, 4, and 6, respectively, and each of the generated data elements may have the same coordinates within the corresponding X, Y, Z coordinate system. In some embodiments, the coordinates of a specific location within the X, Y, Z coordinate system may be referred to as "engine coordinates" or "engine addresses." Although Fig.11 As shown, the multi-engine processor 1123 includes four engines, but in actual implementations, the multi-engine processor 1123 may include any number of engines.

[0112] In some aspects, each of ENG0-ENG3 may use a CNN access mode (as described above for Figures 6A-6D More specifically, each of ENG0-ENG3 can access data from a local memory (e.g., Figure 1 In some embodiments, before ENG0-ENG3 requests the IFM data elements and / or weights used to generate OFM0-OFM7, the IFM data elements and / or weights may be retrieved from the memory (e.g., cache memory 122) according to the CNN access mode. Figure 1 is pre-fetched (or retrieved) from the memory 121 of the system and stored in the local memory (e.g., Figure 1 To facilitate the IFM data elements and / or weights from memory (e.g., Figure 1 The IFM data elements and / or weights may be pre-fetched from the memory 121 according to the CNN access mode (e.g., Figure 1 121) is mapped to the local memory (e.g., Figure 1 cache memory 122).

[0113] Figures 12A-12C1 shows a memory 1221 storing a mapping tensor X according to some embodiments. The mapping tensor X represents a virtually defined tensor that includes one or more data elements stored in the memory 1221 and is configured to map the one or more data elements to a local memory according to a CNN access mode. For simplicity, the one or more data elements and the local memory are referred to in Figures 12A-12C Not shown in . In some aspects, mapping one or more data elements from memory 1221 to local memory according to the CNN access pattern may refer to associating each of the one or more data elements in memory 1221 with a specific location in the local memory, so that if the one or more data elements are transferred or copied from memory 1221 to the local memory, the one or more data elements may be stored in the local memory in the sequence of the CNN access pattern. Further, the one or more data elements may be stored in the local memory so that the one or more data elements are not, for example, a multi-engine processor (e.g., Figure 1 Existing data elements stored in local memory required by the multi-engine processor 123).

[0114] Reference Fig. 12A , the mapping tensor X may include one or more data elements of the sub-tensor X stored in the memory 1221. Further, the mapping tensor X may be used to map one or more data elements of the sub-tensor X to a segment of the cache memory (e.g., Figure 8 The sub-tensor X may be segment 0 of the cache memory 822 of Fig. 9 Subtensor 902 ST0 and the memory 1221 may be Fig. 7A and Figure 7B Further, the CNN access mode may be as described above with reference to Figures 6A-6D Embodiments of CNN access modes discussed. In some embodiments, each of the one or more data elements of the sub-tensor X may represent an IFM (or OFM) data element or a weight.

[0115] like Figures 12A-12C As shown in , each of the mapping tensor X and the sub-tensor X may represent a three-dimensional tensor having width, height, and depth along the X, Y, and Z axes, respectively; however, in actual implementations, each of the mapping tensor X and the sub-tensor X may have any number of dimensions. Further, as Figures 12A-12C As shown in , the spatial dimensions of the mapping tensor X are indicated by the # sign. Fig. 12A, the mapped tensor X includes only one or more data elements along the left portion of the X axis of the sub-tensor X and all data elements along the Y and Z axes of the sub-tensor X, but in some embodiments, the mapped tensor X may include one or more other data elements of the sub-tensor X. For example, Fig. 12B As shown in , the mapped tensor X may include one or more data elements along the middle of the X axis of the sub-tensor X and all data elements along the Y and Z axes of the sub-tensor X. Further, Fig. 12C As shown, the mapped tensor X may include one or more data elements along only the right portion of the X axis and all data elements along the Y and Z axes.

[0116] Fig.13 13 shows a memory 1321 storing a mapping tensor Y according to some embodiments. More specifically, Fig.13 As shown, the mapping tensor Y may include data elements of the sub-tensor Y stored in the memory 1321. The mapping tensor Y, the sub-tensor Y, and the memory 1321 may be Figures 12A-12C An embodiment of a mapping tensor X, a subtensor X, and a memory 1221. Fig.13 As shown in , each of the mapped tensor Y and the subtensor Y has a width and a height along the X and Y axes, respectively. Further, Fig.13 As shown, subtensor Y includes 16 data elements, each of which is labeled with a corresponding number ranging from 1 to 16. However, in actual implementations, subtensor Y may include any number of data elements. In some embodiments, each of the numbers 1-16 may represent a label of a label array. Further, Fig.13 As shown, the mapping tensor Y may include four data elements of the sub-tensor Y - namely, data elements 4, 6, 12, and 15. However, in actual implementations, the mapping tensor Y may include any number of data elements of the sub-tensor Y. In some embodiments, the data elements 4, 6, 12, and 15 of the mapping tensor Y may be placed (or ordered) within the mapping tensor Y according to the CNN access mode.

[0117] Fig.14 14 shows an example system 1400 according to some embodiments, the system comprising a memory 1421 and a cache memory 1422. More specifically, the memory 1421 stores a mapping tensor Z, the mapping tensor comprising data elements of a sub-tensor Z. The mapping tensor Z, the sub-tensor Z, and the memory 1421 may be Fig.13 , and the memory 1321. Further, the cache memory 1422 may be Figure 8 and Fig.10 Embodiments of cache memory 822 and / or 1022.

[0118] like Fig.14As shown in , each of the mapping tensor Z and the sub-tensor Z has a width, height, and depth along the X, Y, and Z axes, respectively. However, in actual implementations, each of the mapping tensor Z and the sub-tensor Z may have any dimensions. Fig.14 It is further shown that the mapping tensor Z includes 8 data elements of the sub-tensor Z. However, in actual implementations, the mapping tensor Z may include any number of data elements of the sub-tensor Z. Fig.14 As shown in , each of the eight data elements is labeled with a corresponding number ranging from 1 to 8. In some embodiments, each of the numbers 1-8 may represent a tag of a tag array.

[0119] Fig.14 As further shown, in some embodiments, data elements 1-4 of the mapping tensor Z may be mapped to cache memory 1422 according to the CNN access mode. More specifically, Fig.14 14 shows that at a given time, data elements 1-4 of the mapped tensor Z may be mapped to segment Z of cache memory 1422 according to the CNN access pattern. That is, data element 1 may be mapped to cache line 1 of segment Z, data element 2 may be mapped to cache line 2 of segment Z, data element 3 may be mapped to cache line 3 of segment Z, and data element 4 may be mapped to cache line 4 of segment Z. In some embodiments, data elements 5-8 of the mapped tensor Z may be similarly mapped to cache lines 1-4 of segment Z, respectively.

[0120] In some aspects, data elements 1-4 of mapped tensor Z may be respectively transferred (or copied) to and stored in cache lines 1-4 of segment Z. When data elements 1-4 of mapped tensor Z are respectively stored in cache lines 1-4 of segment Z, each of data elements 1-4 may replace an existing data element stored in cache lines 1-4 of segment Z. Further, data elements 1-4 respectively stored in cache lines 1-4 of segment Z may then be transferred (or copied) to a multi-engine processor (e.g., Figure 1 The multi-engine processor 123 of segment Z is transmitted to the multi-engine processor for processing according to the CNN access mode. In some embodiments, once the data elements 1-4 respectively stored in the cache lines 1-4 of segment Z are transmitted to the multi-engine processor (or read by the multi-engine processor), the data elements 5-8 of the mapping tensor Z can be respectively transmitted (or copied) to the cache lines 1-4 of segment Z. In other words, data elements 5-8 can overwrite the data elements 1-4 respectively stored in the cache lines 1-4 of segment Z. The data elements 5-8 respectively stored in the cache lines 1-4 of segment Z can then be transmitted to the multi-engine processor (or read by the multi-engine processor) for processing according to the CNN access mode. Although Fig.14 As shown, segment Z includes 4 cache lines, but in actual embodiments, segment Z may include any number of cache lines.

[0121] As mentioned above Fig. 7A and Figure 7B As discussed, the number of sub-tensors associated with a tensor and the dimensions of each of the partitioned sub-tensors may be pre-configured based on a CNN access pattern (e.g., prior to processing of one or more layers of the CNN). Similarly, the dimensions of a mapping tensor (e.g., mapping tensor Z) may be pre-configured based on a CNN access pattern (e.g., prior to processing of one or more layers of the CNN). In some embodiments, pre-configuring (i) the number of sub-tensors into which a tensor is partitioned, (ii) the dimensions of each of the partitioned sub-tensors, and (iii) the dimensions of a mapping tensor associated with each of the partitioned sub-tensors may facilitate the removal of each of the one or more data elements from a memory (e.g., Fig.14 1421) to a cache memory (e.g., Fig.14 mapping of cache memory 1422).

[0122] Fig.15 1521 is shown storing an example tensor 1502 that is virtually divided into multiple (K) sub-tensors according to some embodiments. The tensor 1502 and the memory 1521 may be Figure 7B Further, Fig.15 As shown, tensor 1502 may be a three-dimensional tensor having width, height, and depth along the X, Y, and Z axes, respectively. Further, in some embodiments, tensor 1502 may be virtually divided into K=8 sub-tensors (e.g., 1502 ST0 -1502 ST7 ). In actual implementations, tensor 1502 may have any dimension and be divided into any number of sub-tensors.

[0123] like Fig.15 As shown in , tensor 1502 can be associated with parameters Alloc_Width, Alloc_Height, and Alloc_Depth (where "Alloc" refers to allocation). Parameters Alloc_Width, Alloc_Height, and Alloc_Depth can be collectively referred to as "tensor parameters". Alloc_Width can represent the number of sub-tensors along the X-axis into which tensor 1502 is divided. For example, Alloc_Width = 2 sub-tensors (e.g., 1502 ST2 and 1502 ST3 ),like Fig.15 Alloc_Height may represent the number of sub-tensors along the Y axis into which the tensor 1502 is divided. For example, Alloc_Height = 2 sub-tensors (e.g., 1502 ST5 and 1502ST7 ),like Fig.15 Further, Alloc_Depth may represent the number of sub-tensors along the Z axis into which the tensor 1502 is divided. For example, Alloc_Depth = 2 sub-tensors (e.g., 1502 ST3 and 1502 ST7 ),like Fig.15 In some embodiments, each of the tensor parameters of tensor 1502 may be pre-configured prior to processing of one or more layers of the CNN.

[0124] Fig.16 1602 shows an example sub-tensor stored according to some embodiments ST Memory 1621. Sub-tensor 1602 ST Can be Fig.15 An embodiment of a sub-tensor in a sub-tensor of (e.g., sub-tensor 1502 ST0 ), and the memory 1621 may be Fig.15 An embodiment of the memory 1521. Further, Fig.16 It is shown that subtensor 1602 ST It can be a three-dimensional sub-tensor with width, height and depth along the X, Y and Z axes respectively. However, in actual implementation, the sub-tensor 1602 ST Can have any number of dimensions.

[0125] like Fig.16 As shown in , subtensor 1602 ST Can be associated with parameters SubT_Width, SubT_Height, and SubT_CH_Num (collectively referred to as "sub-tensor parameters"). SubT_Width can represent the sub-tensor 1602 along the X axis ST SubT_Height can represent the sub-tensor 1602 along the Y axis. ST Further, SubT_CH_Num may represent the subtensor 1602 along the Z axis. ST In some embodiments, subtensor 1602 ST Each of the sub-tensor parameters may be pre-configured prior to processing of one or more layers of the CNN.

[0126] Fig.17 1702 shows an example map tensor stored according to some embodiments. MT Memory 1721. Fig.17 As shown in , the mapping tensor 1702 MT May include sub-tensor 1702 ST One or more data values, wherein the mapping tensor 1702 MTand subtensor 1702 ST is stored in the memory 1721. Mapping tensor 1702 MT Can be Fig.14 Further, the sub-tensor 1702 ST and memory 1721 may be Fig.16 Subtensor 1602 ST and memory 1621. In some embodiments, mapping tensor 1702 MT and subtensor 1702 ST Each of may be a three-dimensional array (or matrix) having width, height, and depth along the X, Y, and Z axes, respectively. However, in actual implementations, the mapping tensor 1702 MT and subtensor 1702 ST Each of can have any number of dimensions.

[0127] like Fig.17 As shown in , the mapping tensor 1702MT may be associated with parameters MapT_Width, MapT_Height, and MapT_CH_Num (collectively referred to as "mapping tensor parameters"). MapT_Width may represent the width of the mapping tensor 1702MT along the X-axis. MapT_Height may represent the height of the mapping tensor 1702MT along the Y-axis. Further, MapT_CH_Num may represent the number of channels (or depth) of the mapping tensor 1702MT along the Z-axis. In some embodiments, the mapping tensor 1702 MT Each of the mapping tensor parameters may be pre-configured prior to processing of one or more layers of the CNN.

[0128] In some embodiments, the tensor parameters, subtensor parameters, and mapped tensor parameters associated with a given data element of a tensor (see above) are Figure 15-17 discussed) can facilitate the transfer of data elements from memory (e.g., Figure 1 Memory 121) to local memory (e.g., Figure 1 In other words, the tensor parameters, sub-tensor parameters, and mapping tensor parameters associated with a given data element may be used to determine where the data element may be mapped to (or stored in) local memory. Similarly, the engine address (e.g., (X, Y, Z)) associated with the data element (e.g., the engine address at which the data element may be processed by the engine) may also be used to determine where the data element may be stored in local memory.

[0129] Fig.18 1800 according to some embodiments. The cache address 1800 may represent a cache memory (e.g., Fig.10The virtual address or location of a data element stored in the cache memory 1022). Fig.18 As shown in , cache address 1800 may include a tag, a segment number (also referred to as "segment_num"), an index, and an offset. In some embodiments, tensor parameters, subtensor parameters, mapping tensor parameters, and engine addresses associated with a given data element may be used to determine the cache address (e.g., index, segment number, offset, and tag) of the data element. The index of cache address 1800 refers to a specific cache row of cache memory (e.g., Fig.14 To determine the index of a data element, a mapping address (also referred to as "map_address") is first calculated based on the engine address associated with the data element (e.g., (X, Y, Z)), the subtensor parameters, and the mapping tensor parameters in Equation 1 as follows: Map_Address=(X%SubT_Width)%MapT_Width+MapT_Width*((Y%SubT_Height)%MapT_Height)+MapT_Width*MapT_Height*((Z%SubT_CH_Num)%MapT_Ch_Num)

[0130] In Equation 1, the symbol % represents a modulo operator. In some aspects, the subtensor parameter of Equation 1 may be associated with a subtensor of at least three dimensions and a tensor of at least three dimensions, wherein the tensor includes the subtensor. Similarly, the mapping tensor parameter of Equation 1 may be associated with a mapping tensor of at least three dimensions.

[0131] In some embodiments, after determining the mapping address of the data element, the index may be calculated in Equation 2 as follows:

[0132] In Equation 2, section_size represents the size of the segment in the cache memory (as described above with reference to Fig.10 The term "floor" refers to a floor function.

[0133] In some embodiments, the segment number of the cache address 1800 of a data element refers to a number reflecting the particular segment in which the data element is stored (e.g., Fig.10 The segment number is "1" in the Num ) can be based on the engine coordinates, subtensor parameters, and tensor parameters associated with the data elements in the Formula 3 is determined as follows:

[0134] In Equation 3, the term "floor" represents a floor function, as explained above with reference to Equation 2.

[0135] In some embodiments, the offset of cache address 1800 refers to the location of a data element within a segment of a segment of cache memory. The offset can be determined in Equation 4 based on the mapping address of Equation 1 and the segment size (also referred to as "section_size") as follows: Offset=Map_Address%Section_Size

[0136] In Equation 4, the symbol % represents the modulo operator, as explained above.

[0137] To determine the tag of the cache address 1800 of a data element, the segment address (also referred to as “segment_address”) may first be calculated based on the engine address and subtensor parameters associated with the data element in Equation 5 as follows: Segment Address = X% SubT Width +(Y%SubT Height )*SubT Width +(Z%SubT_CH_Num)*SubT_Width*SubT_Height In Equation 5, the symbol % represents the modulo operator, as explained above.

[0138] In some aspects, the tag of cache address 1800 represents a unique identification of a data element stored in the cache memory. In some embodiments, the tag may be determined in Equation 6 based on the segment address and segment size of Equation 5 as follows:

[0139] In Equation 6, the term "floor" represents the floor function, as discussed above.

[0140] Fig.19 1900 according to some embodiments. In some aspects, the tag array 1900 can be processed by a data processing system (e.g., Figure 1 data processing system 120) to manage cache memory (e.g., Figure 1 In the case where the cache memory stores L data elements, a tag (e.g., tag 1, tag 2, . . . , tag L) of each corresponding data element in the L data elements may be stored in the tag array 1900. In some aspects, each tag of the tag array 1900 may be associated with the cache memory (e.g., Figure 1The tag array 1900 may also store a corresponding valid bit for each segment of the cache memory for each of tag 1, tag 2, ... tag L. For example, for Fig.19 1, the tag array 1900 includes the following valid bits: 段1 , effective position 段2 , ..., effective position 段K In some embodiments, each valid bit of a corresponding segment may indicate whether a data element of a corresponding tag is currently stored in the corresponding segment. Fig.19 Tag 1, valid bit 区段1 The valid bit may indicate whether the data element identified by tag 1 is currently stored in segment 1 of the cache memory. In some embodiments, the valid bit (for the corresponding segment) may be set to a value of 1 when the data element of the tag corresponding to the valid bit is currently stored in the corresponding segment; otherwise, the valid bit may be set to a value of 0. In some aspects, one or more tags of tag array 1900 may respectively identify a specific (or the same) segment of the cache memory (e.g., Fig.10 One or more data elements stored in segment 1) of cache memory 1022.

[0141] As mentioned above Fig.11 As described, when multiple engines process layers of a CNN, the engines may operate in lock step. In some embodiments, to facilitate this operation, and to ensure that the engines do not suffer contention when reading data elements stored in one or more segments, the engines may be subject to various restrictions.

[0142] Fig. 20 1 shows a set of engines (e.g., ENG0-ENG7) subject to various restrictions when processing layers of a CNN according to some embodiments. More specifically, Fig. 20 ENG0-ENG7 are shown to be allocated according to the combined allocation, and during a given time period, each of ENG0-ENG7 can produce an OFM data element located at, for example, the origin of the corresponding X, Y, Z coordinate system (e.g., an OFM data element having engine coordinates of (0, 0, 0)). In other words, during a given time period, each of ENG0-ENG7 can convolve the IFM data elements from the cache memory to produce a corresponding OFM data element having engine coordinates of (0, 0, 0).

[0143] In some embodiments, to ensure that there is no read contention between each of ENG0-ENG7, the distance between each of the OFM data elements having engine coordinates of (0,0,0) may be limited. Fig. 20For ENG0 and ENG1, the distance between the OFM data element generated by ENG0 at (0, 0, 0) and the OFM data element generated by ENG1 at (0, 0, 0) along the X-axis can be limited to i*SubT_Width, where i represents an integer (e.g., i=...-3, -2, -1, 0, 1, 2, 3...). SubT_Width can represent a sub-tensor parameter of a sub-tensor including (one or more) IFM data elements used to generate OFM data elements at (0, 0, 0) for each of ENG0 and ENG1. Similarly, the distance between the OFM data element generated by ENG0 at (0, 0, 0) and the OFM data element generated by ENG2 at (0, 0, 0) along the Y-axis can be limited to j*SubT_Height, where j represents an integer (e.g., j=...-3, -2, -1, 0, 1, 2, 3...). SubT_Height may represent a subtensor parameter of a subtensor including (one or more) IFM data elements used to generate an OFM data element at (0, 0, 0) for each of ENG0 and ENG2. Similarly, the distance between the OFM data element generated by ENG0 at (0, 0, 0) and the OFM data element generated by ENG4 at (0, 0, 0) may be limited to k*SubT_CH_Num along the Z-axis, where k represents an integer (e.g., j=...-3, -2, -1, 0, 1, 2, 3...). SubT_CH_Num may represent a subtensor parameter of a subtensor including (one or more) IFM data elements used to generate an OFM data element at (0, 0, 0) for each of ENG0 and ENG4.

[0144] Fig.21 2100 according to some embodiments. Figure 1 In some embodiments, data processing system 2100 may represent a neural network processing system (e.g., a convolutional neural network processing system). Fig.21 As shown in , in some embodiments, the data processing system 2100 may include a memory 2121, a cache memory 2122, a multi-engine processor 2123, and a plurality of (K) multiplexers (e.g., MUX 1, MUX 2, ..., MUX K). The data processing system 2100 may also include a pre-fetch address generator 2125, a tag array 2126, a load queue 2127, an external read address generator 2128, a task queue 2129, and an address generator 2130. The memory 2121, the cache memory 2122, and the multi-engine processor 2123 may be Figure 1In some embodiments, the data processing system 2100 may be configured to execute CNN using, for example, the multi-engine processor 2123 and the cache memory 2122.

[0145] In some embodiments, memory 2121 may represent main memory, shared memory, or external memory. In embodiments where memory 2121 represents external memory, memory 2121 may be separate from (or external to) data processing system 2100. Further, in some embodiments, memory 2121 may store one or more tensors of data elements (e.g., Fig. 7A and Figure 7B 702A and / or 702B. For example, the memory 2121 may store one or more tensors of weights, one or more tensors of IFM data elements, and / or one or more tensors of OFM data elements. In some embodiments, each of the one or more tensors stored in the memory 2121 may be virtually divided into multiple (K) sub-tensors. Further, in some embodiments, the memory 2121 may be configured to receive one or more commands from the external read address generator 2128. The memory 2121 may also be configured to use, for example, a mapping tensor (e.g., Fig.14 The mapped tensor Z) outputs one or more data elements to the cache memory 2122.

[0146] like Fig.21 As shown in , the cache memory 2122 may include multiple (K) segments. Each of the K segments may be configured to store one or more data elements of a tensor that is divided into K sub-tensors and stored in the memory 2121. More specifically, each of the K segments may be configured to store one or more data elements of a corresponding one of the K divided sub-tensors (of a tensor) stored in the memory 2121. Further, in some embodiments, the cache memory 2122 may be configured to store only one type of data element (at least at a given time). For example, the cache memory 2122 may be configured to store only IFM data elements, only weights, or only OFM data elements.

[0147] Although Fig.21Only one cache memory 2122 is shown, but in some embodiments, the data processing system 2100 may include multiple cache memories 2122. For example, the data processing system 2100 may include one or more cache memories 2122 configured to store IFM data elements and / or one or more cache memories 2122 configured to store weights and / or one or more cache memories 2122 configured to store OFM data elements. In some embodiments, the cache memory 2122 may represent a level 1 (also referred to as "L1") cache memory. Further, the cache memory 2122 may be configured to output one or more data elements and receive one or more data elements from the multi-engine processor 2123. The cache memory 2122 may also be configured to receive one or more commands from the address generator 2130.

[0148] In some embodiments, each of the K multiplexers (e.g., MUX 1, MUX 2, ..., MUX K) may be configured to select a corresponding segment of the cache memory 2122 based on the segment number determined using Equation 3. Further, each of the K multiplexers may be configured to receive one or more data elements from the corresponding selected segment and output the one or more data elements to a corresponding engine of the multi-engine processor 2123. In some embodiments, each of the K multiplexers may also be configured to receive one or more commands from the address generator 2130.

[0149] like Fig.21 As shown in , the multi-engine processor 2123 may include multiple (K) engines (e.g., engine 1, engine 2, ..., engine K). In some aspects, each of the K engines may represent a corresponding processor or core, and is Fig. 20 In some embodiments, the multi-engine processor 2123 may represent a neural network processor (e.g., a convolutional neural network processor) that is configured to use K engines to execute, for example, a CNN and output one or more inferences or predictions. In such embodiments, each of the K engines may operate in lock step to process a given layer of the CNN according to a CNN access pattern. Further, the operations of the K engines (e.g., load, multiply, and / or accumulate operations) may be determined (or predicted) before executing the CNN. In some other embodiments, the multi-engine processor 2123 may represent a neural network processor that is configured to use K engines to train and output a neural network model. Fig.21 As shown in , in some embodiments, the multi-engine processor 2123 may be configured to receive one or more commands from the task queue 2129. Further, the multi-engine processor 2123 may be configured to generate one or more requests for data elements (e.g., IFM or OFM data elements or weights) required to execute, for example, a CNN.

[0150] The controller 2124 may represent a sequencer and / or a central processing unit configured to manage the data processing system 2100. In some embodiments, before the execution of, for example, a CNN, the controller 2124 may determine the operations (e.g., load, multiply, and / or accumulate operations) that the multi-engine processor 2123 will perform during the execution of the CNN. Further, the controller 2124 may output commands related to the operation of the multi-engine processor 2123. In some embodiments, the controller 2124 may include a pre-fetch address generator 2125 and / or an address generator 2130.

[0151] In some embodiments, the pre-fetch address generator 2125 may be configured to receive commands from the controller 2124. The pre-fetch address generator 2125 may also be configured to use the received commands to generate engine coordinates (e.g., (X, Y, Z)) for each of the one or more data elements (e.g., IFM data elements, OFM data elements, and / or weights) that each of Engines 1-Engine K will need to produce one or more output data elements (e.g., OFM data elements). For example, in some embodiments, the pre-fetch address generator 2125 may generate engine coordinates (0, 0, 0), which may refer to the origin of the X, Y, Z coordinate system of Engine 1 of the multi-engine processor 2123. The engine coordinates (0, 0, 0) may also correspond to the IFM data element that Engine 1 will need to process in order to produce the OFM data element located at (0, 0, 0) of the X, Y, Z coordinate system of Engine 1.

[0152] In some embodiments, the pre-fetch address generator 2125 may generate one or more engine coordinates (or engine addresses) for one or more data elements many clock cycles before when Engine 1-Engine K may request the one or more data elements. Accordingly, the pre-fetch address generator may generate the engine coordinates for the one or more data elements so as to pre-fetch the one or more data elements from the memory 2121 for storage in the cache memory 2122 (before Engine 1-Engine K requests the one or more data elements). Further, in some embodiments, the pre-fetch address generator 2125 may translate each of the one or more engine coordinates into a corresponding cache address (including, for example, a tag, a segment number, an index, and an offset, as described above for Fig.18 The pre-fetch address generator 2125 may also generate and output one or more commands, each of which may include the translated cached addresses of one or more data elements.

[0153] The tag array 2126 may be Fig.192125 and may be configured to receive one or more commands from the pre-fetch address generator 2125. In some embodiments, for a given command received from the pre-fetch address generator 2125, the tag array 2126 may be configured to compare the cache address (for a particular data element) included in the command with the contents of the tag 2126 (e.g., the tag and valid bits) to determine whether the data element is currently stored in the cache memory 2122. In some instances, the tag array 2126 may determine that the data element corresponding to the cache address is not currently stored in the cache memory 2122 because the data element has not been requested by the multi-engine processor 2123 or the address generator 2130 and is therefore being pre-fetched. This determination may be referred to as a "cache miss". When there is a cache miss, the tag array 2126 may generate a request (or command) for the data element to be fetched from the memory 2121, wherein the request may include the engine coordinates of the data element.

[0154] In some embodiments, the request may also include the engine coordinates of one or more other data elements that will be stored near the data element in the cache memory 2122 based on the principle of spatial location (and according to the CNN access pattern). For example, the request may include not only the engine coordinates of the data element, but also the engine coordinates of all elements stored in the same cache line as the data element in the cache memory 2122 based on the CNN access pattern. In the case where the tag array 2126 determines that the data element corresponding to the cache address is currently stored in the cache memory 2122, the tag array 2126 may not generate a request for the data element. Further, this determination may be referred to as a "cache hit".

[0155] In some embodiments, load queue 2127 may represent a buffer (e.g., a first-in, first-out buffer) configured to store one or more requests received from tag array 2126. In some other embodiments, load queue 2127 may represent a linked list that includes one or more requests received from tag array 2126. Load queue 2127 may be configured to output one or more requests.

[0156] In some embodiments, the external read address generator 2128 may be configured to receive one or more requests from the load queue 2127. The external read address generator 2128 may also be configured to translate each of the engine coordinates included in the one or more requests into a corresponding address within the memory 2121. For example, the external read address generator 2128 may be configured to translate the engine coordinates of a data element into an address of the data element in the memory 2121.

[0157] In some embodiments, the address in memory 2121 of a data element may be based on a tensor (e.g., Fig. 7A The address in memory 2121 of the data element may also be determined based on various dimensions of the tensor, including the width of the tensor (also referred to as "MainT_Width"), the height of the tensor (also referred to as "MainT_Height"), and the depth of the tensor (also referred to as "MainT_CH_Num"). In some embodiments, MainT_Width, MainT_Height, and MainT_CH_Num may be expressed in units of bytes or words. Accordingly, the external read address generator 2128 may translate the engine coordinates of the data element into an address in memory 2121 as follows in Equation 7:

[0158] Once the external read address generator 2128 has translated the engine coordinates of each data element in the one or more requests into a corresponding address in the memory 2121, the external read address generator 2128 may proceed to generate one or more read requests for the data elements. Each read request may include, for example, an address in the memory 2121 of at least one data element. In some embodiments, the memory 2121 may receive the one or more read requests and use the one or more addresses in the one or more read requests to pre-fetch (or fetch) the corresponding one or more data elements. The one or more data elements may then be read using a mapping tensor (e.g., Fig.14 The mapped tensor Z) is transferred (or copied) from the memory 2121 to the cache memory 2122 and stored in the cache memory 2122. Because one or more data elements may be pre-fetched from the memory 2121 and then stored in the cache memory 2122 according to the CNN access pattern, the data elements may be easily available for processing by the multi-engine processor 2123. Therefore, the multi-engine processor 2123 may not suffer from idle cycles (or delays) resulting from cache misses.

[0159] In some embodiments, the task queue 2129 may be a buffer (e.g., a first-in, first-out buffer) configured to store one or more commands received from the controller 2124. In some other embodiments, the task queue 2129 may be a linked list that includes one or more commands received from the controller 2124. The task queue 2129 may be configured to output the one or more commands to the address generator 2130 and / or the multi-engine processor 2123. In some embodiments, in the event that the multi-engine processor 2123 has generated a request for a data element that is not currently stored in the cache memory 2122, the task queue 2129 may not output the one or more commands to the multi-engine processor 2123 (or may hold the one or more commands) until the multi-engine processor 2123 receives the requested data element.

[0160] The address generator 2130 may also be configured to receive one or more commands from the task queue 2129. The address generator 2130 may also be configured to use the received commands to generate engine coordinates (e.g., (X, Y, Z)) for each of the one or more data elements that each of Engines 1-Engine K currently needs in order to produce one or more OFM data elements. Further, in some embodiments, the address generator 2130 may receive one or more requests for one or more data elements from the multi-engine processor 2123, wherein each of the one or more requests includes the engine coordinates of the requested one or more data elements.

[0161] The address generator 2130 may be further configured to translate the engine coordinates received from the multi-engine processor 2123 and / or generated by the address generator 2130 into one or more cache addresses, wherein each cache address corresponds to a corresponding data element. Further, each cache address may include, for example, a tag, a segment number, an index, and an offset, as described above with respect to Fig.18 In some embodiments, the address generator 2130 may also generate and output one or more commands, each of which may include one or more cache addresses. The address generator 2130 may further output the one or more commands to a multiplexer (e.g., MUX 1, MUX 2, ..., MUXK) and / or a cache memory 2122. In some embodiments, the multiplexer (e.g., MUX 1, MUX 2, ..., MUXK) and / or the cache memory 2122 may use the cache addresses in the one or more commands to fetch the corresponding data elements, and output the data elements to the multi-engine processor 2123.

[0162] Fig. 22 2200 according to some embodiments. The data processing system 2200 may be Fig.21In some embodiments, the data processing system 2200 may be configured to execute a neural network (e.g., CNN) using a multi-engine processor 2230 and a cache memory 2257. Fig. 22 As shown in FIG. 2 , a data processing system 2200 may include a device interface (“I / F”) 2210, a multi-engine processor 2230, a memory 2240, and a storage 2250. In some embodiments, the device I / F 2210 may include an input I / F 2211 configured to receive input data 2201 (e.g., Figure 1 The input I / F 2211 may also be configured to communicate with the memory 2240, the multi-engine processor 2230, and / or the memory 2250. Fig. 22 As shown in FIG. 2 , the device I / F 2210 may also include an output I / F 2212 configured to output (or transmit) output data 2203 (e.g., Figure 1 The output I / F 2212 may be configured to communicate with the memory 2240, the multi-engine processor 2230 and / or the memory 2250.

[0163] In some embodiments, the multi-engine processor 2230 may be Fig.21 In some embodiments, the multi-engine processor 2230 may include multiple (K) engines. Further, the multi-engine processor 2230 may represent a neural network processor (e.g., a convolutional neural network processor) configured to execute a neural network (e.g., CNN) of a neural network SW module 2259 of the memory 2250. The operations performed by the K engines of the multi-engine processor 2230 may be known (or determined) before the neural network is executed. In some embodiments, the multi-engine processor 2230 may be configured to receive one or more commands from a task queue 2256 of the memory 2250. The multi-engine processor 2230 may also be configured to generate one or more requests for data elements required to execute the neural network.

[0164] In some embodiments, the memory 2240 may include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, Flash memory, hard disk drive, and the like) that can store input data 2201. The memory 2240 may represent a main memory, a shared memory, or an external memory. Further, in some aspects, the memory 2240 may represent a normal linear memory. In an embodiment in which the memory 2240 represents an external memory, the memory 2240 may be separated from the data processing system 2200 (or external to the data processing system 2100). In some embodiments, the memory 2240 may be configured to receive configuration data (e.g., tensor parameters, sub-tensor parameters, and / or mapping tensor parameters) from the memory 2250. The memory 2240 may also be configured to virtually divide at least one tensor stored in the memory 2240 into a plurality of (K) sub-tensors using the configuration data. Further, the memory 2240 may be configured to receive one or more commands from the external read address generator SW module 2255 of the memory 2250. In some embodiments, the memory 2240 may also be configured to output one or more data elements to the cache memory 2257 of the memory 2250 using a mapping tensor defined by the configuration data.

[0165] The memory 2250 may include a non-transitory computer-readable medium (including one or more non-volatile memory elements such as EPROM, EEPROM, Flash memory, hard drive, and the like) that may store at least the following items: Controller SW module 2251 configured to (i) manage data processing system 2200, (ii) determine, prior to execution of a neural network (e.g., CNN), operations to be performed by multi-engine processor 2230 during execution of the neural network; and (iii) output commands related to the determined operations; a pre-fetch address generator SW module 2252 configured to receive a command from the controller SW module 2251, generate engine coordinates corresponding to one or more data elements associated with the command, translate the engine coordinates into one or more cache addresses, and output a command including at least the engine coordinates and the one or more cache addresses; a tag array 2253 configured to receive one or more commands (including the engine coordinates and the one or more cache addresses) from the pre-fetch address generator SW module 2252, determine whether one or more data elements corresponding to the one or more cache addresses are currently stored in the cache memory 2257, and output one or more requests including at least the cache addresses and engine coordinates of one or more data elements not currently stored in the cache memory 2257; A load queue 2254 configured to store one or more requests received from the tag array 2253 and output one or more requests; an external read address generator SW module 2255 configured to receive one or more requests from the load queue 2254, translate each of the engine coordinates included in the one or more requests into a corresponding address within the memory 2240, and output one or more read requests including at least the translated addresses; A task queue 2256 configured to store one or more commands received from the controller SW module 2251 and output one or more commands; Cache memory 2257 configured to (i) receive and store one or more data elements from memory 2240, (ii) receive one or more commands from address generator SW module 2258, (iii) receive and store one or more data elements from multi-engine processor 2230, and (iv) output one or more data elements; an address generator SW module 2258 configured to receive one or more commands from the task queue 2256 and / or one or more requests from the multi-engine processor 2230 and output one or more commands requesting one or more data elements; and Neural network SW module 2259 , including a neural network model (e.g., CNN) configured to be executed by the multi-engine processor 2230 .

[0166] Each software module includes instructions that, when executed by the data processing system 2200, cause the data processing system 2200 to perform corresponding functions.

[0167] The data processing system 2200 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the data processing system 2200 (such as in the memory 2250). For example, the data processing system 2200 may execute the controller SW module 2251 to (i) manage the data processing system 2200, (ii) determine, before the execution of the neural network (e.g., CNN), the operations to be performed by the multi-engine processor 2230 during the execution of the neural network; and (iii) output at least one command related to the determined operations.

[0168] In some embodiments, the data processing system 2200 may further execute the pre-fetch address generator SW module 2252 to receive a command from the controller SW module 2251, generate engine coordinates of a data element associated with the received command, translate the engine coordinates into a cache address of the data element, and output a command including at least the engine coordinates and the cache address of the data element. The data processing system 2200 may further use the tag array 2253 to (i) receive a command from the pre-fetch address generator SW module 2252, (ii) determine whether the data element corresponding to the cache address in the command is currently stored in the cache memory 2257, and (iii) if the data element is not currently stored in the cache memory 2257, output a request including at least the engine coordinates and the cache address of the data element.

[0169] In some embodiments, the data processing system 2200 may use the load queue 2254 to store the requests received from the tag array 2253 and output the requests. The data processing system 2200 may further execute the external read address generator SW module 2255 to receive the request, translate the engine coordinates (for the data elements) in the request into addresses in the memory 2240, and output the read request including the translated addresses in the main memory 2240. In some embodiments, the data processing system 2200 may use the memory 2240 to receive the request from the external read address generator SW module 2252, use the mapping tensor (e.g., Fig.14 The data processing system 2200 may use the multi-engine processor 2230 to pre-fetch (or fetch) the data elements stored at the translated addresses in the memory 2240. In some embodiments, before the data elements are pre-fetched (or fetched) from the memory 2240, the data elements may have been input to the input I / F 2211 (e.g., as input data 2201) and transferred from the input I / F 2211 to the memory 2240 for storage. The data processing system 2200 may further use the cache memory 2257 to receive the data elements from the memory 2240, store the data elements, and output the data elements to the multi-engine processor 2230. In some embodiments, the data processing system 2200 may use the multi-engine processor 2230 and the data elements output to the multi-engine processor 2230 to execute the neural network model of the neural network SW module 2259. The data processing system 2200 may then generate OFM data elements based at least in part on the data elements and the neural network model. In some embodiments, the data processing system 2200 may store the OFM data elements in the cache memory 2257 and / or the memory 2240. The data processing system 2200 may further transmit the OFM data element to the output I / F 2212, and in some embodiments, the output I / F 2212 may transmit (or output) the OFM data element as output data 2203, such as Fig. 22 as shown in .

[0170] Further, in some embodiments, the data processing system 2200 may use the task queue 2256 to receive commands from the controller SW module 2251 and output the commands to the address generator SW module 2258 and / or the multi-engine processor 2230. The data processing system 2200 may execute the address generator SW module 2258 to receive commands from the task queue 2256. In some embodiments, the data processing system 2200 may further execute the address generator SW module 2258 to output commands (e.g., requesting one or more data elements) to the cache memory 2257.

[0171] Fig.23 An illustrative flow chart showing example operations 2300 is shown in accordance with some embodiments. Example operations 2300 may be performed by a data processing system (e.g., Fig.21 and Fig. 22 2100 and / or 2200) to execute using a neural network processor (e.g., Fig.21 Multi-engine processor 2123) and cache memory ( Fig.21 In some embodiments, the data processing system may include at least a neural network processor and a cache memory. Further, in some embodiments, the data processing system may include a memory (e.g., Fig.21 2121), while in other embodiments the memory may be external to the data processing system.

[0172] like Fig.23 As shown in , a data processing system may divide a tensor including a plurality of data values ​​into a plurality of (K) sub-tensors, wherein each of the K sub-tensors includes a corresponding subset of the plurality of data values ​​(2310). In some embodiments, the tensor and the K sub-tensors may be stored in a memory (e.g., Fig.21 2121). Further, multiple data values ​​may collectively represent, for example, an IFM, an OFM, or multiple weights (e.g., of a CNN model). In some embodiments, each of the K sub-tensors may have the same size or dimension (e.g., having the same width, height, and depth).

[0173] The data processing system may further retrieve one or more first data values ​​of a subset of the data values ​​included in a first sub-tensor of the K sub-tensors according to an access pattern associated with the neural network processor (2320). The one or more first data values ​​may be Fig.14 The access mode can be as shown above. Figures 6A-6D Embodiments of the CNN access mode discussed, and the neural network processor may be Fig.21In some embodiments, the one or more first data values ​​of the subset may be retrieved from a memory (e.g., Fig.21 Further, each of the one or more first data values ​​of the subset may be associated with a corresponding cache address including a tag, a segment number, an index, and an offset. Each corresponding cache address may be the above reference Fig.18 Each of the one or more first data values ​​of the subset may be associated with a processor address comprising a plurality of coordinates of the coordinate system. Each of the processor addresses may be the above referenced Fig.11 Further, in some embodiments, the data processing system can retrieve the one or more first data values ​​of the subset by pre-fetching the one or more first data values ​​of the subset before receiving a request for the one or more first data values ​​of the subset from the neural network processor.

[0174] In some embodiments, the data processing system may store one or more first data values ​​of the subset of data values ​​in one of K segments of a cache memory, wherein each of the K segments may be associated with a corresponding one of the K sub-tensors (2330). The cache memory may include a level 1 (L1) cache memory. Further, the one or more first data values ​​of the subset may be associated with multiple dimensions of a mapping tensor, wherein the mapping tensor is configured to map the subset of data values ​​included in the first sub-tensor to a segment of the cache memory associated with the first sub-tensor based on an access pattern. The mapping tensor may be Fig.14 The mapping tensor Z and / or Fig.17 The mapping tensor 1702 MT Further, the multiple dimensions of the mapping tensor may include the width, height, and channel of the mapping tensor. The width, height, and channel of the mapping tensor may be the same as those in the above reference. Fig.17 Examples of the parameters MapT_Width, MapT_Height and MapT_CH_Num are discussed.

[0175] The data processing system may further use the neural network processor to process one or more first data values ​​of the subset of data values ​​according to the access pattern (2340). In some embodiments, the neural network processor includes K processors, wherein each of the K processors is configurable to process data using the coordinate system. The K processors of the neural network processor may be the above referenced Fig.21 An example of K engines of the multi-engine processor 2123 is discussed. Further, the neural network processor may represent a convolutional neural network processor.

[0176] In some embodiments, the data processing system may further retrieve one or more second data values ​​of a subset of the data values ​​included in the first sub-tensor based on the access mode, wherein the one or more second data values ​​of the subset are associated with multiple dimensions of the mapped tensor. The one or more second data values ​​may be Fig.14 Data elements labeled 5-8 and / or Fig. 12B and / or Fig. 12C The data processing system may further store one or more second data values ​​of the subset in a segment of a cache memory associated with the first sub-tensor. Additionally, the data processing system may use a neural network processor to process the one or more second data values ​​of the subset according to the access pattern.

[0177] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0178] In addition, those skilled in the art will appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above according to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. A skilled person can implement the described functionality in a variety of ways for each specific application, but this type of implementation determination should not be interpreted as causing a departure from the scope of the present disclosure.

[0179] The methods, sequences or algorithms described in conjunction with the aspects disclosed herein may be implemented directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated into the processor.

[0180] In the foregoing description, embodiments have been described with reference to specific examples thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the present disclosure as set forth in the appended claims. The specification and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method comprising: Partitioning a tensor comprising a plurality of data values ​​into a plurality of (K) sub-tensors, wherein each of the K sub-tensors comprises a respective subset of the plurality of data values; retrieving one or more first data values ​​of the subset of data values ​​included in a first sub-tensor of the K sub-tensors according to an access pattern associated with a neural network processor; storing the one or more first data values ​​of the subset of data values ​​in one of K segments of a cache memory, wherein each segment of the K segments is associated with a respective one of the K sub-tensors; as well as The one or more first data values ​​of the subset of data values ​​are processed using the neural network processor according to the access pattern.

2. The method of claim 1, wherein: Each first data value of the one or more first data values ​​of the subset is associated with a corresponding cache address including a tag, a segment number, an index, and an offset.

3. The method of claim 1, wherein: Each first data value of the one or more first data values ​​of the subset is associated with a processor address comprising a plurality of coordinates of a coordinate system.

4. The method of claim 3, wherein: The neural network processor comprises K processors, and wherein each of the K processors is configured to process data using the coordinate system.

5. The method of claim 1, wherein: The one or more first data values ​​of the subset are associated with multiple dimensions of a mapping tensor, wherein the mapping tensor is configured to map the subset of data values ​​included in the first sub-tensor to the segment of the cache memory associated with the first sub-tensor according to the access pattern.

6. The method of claim 5, wherein: The multiple dimensions include a width, a height, and channels of the mapping tensor.

7. The method of claim 5, further comprising: retrieving one or more second data values ​​of the subset of data values ​​included in the first sub-tensor according to the access pattern, wherein the one or more second data values ​​of the subset are associated with the plurality of dimensions of the mapped tensor; storing the one or more second data values ​​of the subset in the segment of the cache memory associated with the first sub-tensor; as well as The one or more second data values ​​of the subset are processed using the neural network processor according to the access pattern.

8. The method of claim 1, further comprising: A request for the one or more first data values ​​of the subset is received from the neural network processor, wherein retrieving the one or more first data values ​​of the subset includes pre-fetching the one or more first data values ​​of the subset prior to receiving the request from the neural network processor.

9. The method of claim 1, wherein: Retrieving the one or more first data values ​​of the subset includes retrieving the one or more first data values ​​of the subset from a memory.

10. The method of claim 1, wherein: The multiple data values ​​represent an input feature map, an output feature map, or a plurality of weights.

11. The method of claim 1, wherein: Each of the K sub-tensors has the same size or dimension.

12. The method of claim 1, wherein: The cache memory includes a level 1 (L1) cache memory.

13. The method of claim 1, wherein: The neural network processor includes a convolutional neural network processor.

14. A system comprising: Processing systems; as well as a memory storing instructions that, when executed by the processing system, cause the system to: Partitioning a tensor comprising a plurality of data values ​​into a plurality of (K) sub-tensors, wherein each of the K sub-tensors comprises a respective subset of the plurality of data values; retrieving one or more first data values ​​of the subset of data values ​​included in a first sub-tensor of the K sub-tensors according to an access pattern associated with a neural network processor; storing the one or more first data values ​​of the subset of data values ​​in one of K segments of a cache memory, wherein each segment of the K segments is associated with a respective one of the K sub-tensors; as well as The one or more first data values ​​of the subset of data values ​​are processed using the neural network processor according to the access pattern.

15. The system of claim 14, wherein: Each first data value of the one or more first data values ​​of the subset is associated with a corresponding cache address including a tag, a segment number, an index, and an offset.

16. The system of claim 14, wherein: Each of the one or more first data values ​​of the subset is associated with a processor address comprising a plurality of coordinates of a coordinate system.

17. The system of claim 16, wherein: The neural network processor comprises K processors, and wherein each of the K processors is configured to process data using the coordinate system.

18. The system of claim 14, wherein: The one or more first data values ​​of the subset are associated with multiple dimensions of a mapping tensor, wherein the mapping tensor is configured to map the subset of data values ​​included in the first sub-tensor to the segment of the cache memory associated with the first sub-tensor according to the access pattern.

19. The system of claim 14, wherein: Execution of the instructions further causes the system to: A request for the one or more first data values ​​of the subset is received from the neural network processor, wherein retrieving the one or more first data values ​​of the subset includes pre-fetching the one or more first data values ​​of the subset prior to receiving the request from the neural network processor.

20. The system of claim 14, wherein: Each of the K sub-tensors has the same size or dimension.

Images