Neural network processor and system-on-a-chip, data processing method and storage medium

Abstract

This application discloses a neural network processor, a system-on-a-chip, a data processing method, and a storage medium, relating to the system-on-a-chip technology field. [Solution] The neural network processor includes a first processor core comprising: a first cache memory for caching a first input tensor corresponding to a first neural network layer in a neural network model; a first direct memory access controller for reading a second input tensor corresponding to the first neural network layer from a second cache memory and writing the second input tensor to an operational array; and an operational array for reading a first input tensor from the first cache memory, performing a first operation based on the first input tensor and the second input tensor, and obtaining a first output tensor.

Description

Technical Field

[0001] The present disclosure relates to the field of system-on-chip technology, and particularly to neural network processors and system-on-chips, data processing methods, and storage media.

Background Art

[0002] Generally, a neural network processor of a system-on-chip (SoC) processes a relatively complex neural network model. When there are many neural network layers processed by the processor core in the neural network processor, the central processing unit (CPU) issues commands frequently to read weight parameters and / or data to be processed from a lower-level cache memory in the processor core or a memory outside the processor core, and perform operations on the read weight parameters and / or data to be processed. This data reading method that frequently issues CPU commands not only frequently occupies the CPU bus bandwidth, increasing the occupancy of the CPU bus bandwidth and the system power consumption, but also reduces the calculation efficiency of the neural network processor in processing the neural network model.

Summary of the Invention

Problems to be Solved by the Invention

[0003] When reading weight parameters and / or some data to be processed by a CPU command-based data reading method, it not only frequently occupies the CPU bus bandwidth, increasing the occupancy of the CPU bus bandwidth and the system power consumption, but also reduces the calculation efficiency of the neural network processor in processing the neural network model.

Means for Solving the Problems

[0004] To solve the above technical problems, this disclosure provides a neural network processor, which includes a first processor core, A first cache memory used to cache the first input tensor corresponding to the first neural network layer in a neural network model, A first direct memory access controller is used to read the second input tensor corresponding to the first neural network layer from the second cache memory and write the second input tensor to the operational array, The system includes an operation array used to read a first input tensor from a first cache memory, and to perform a first operation corresponding to a first neural network layer based on the first and second input tensors, in order to obtain a first output tensor.

[0005] A second aspect of this disclosure provides a system-on-chip including on-chip memory, a second direct memory access controller, and a neural network processor, wherein the neural network processor includes at least one processor core, and the at least one processor core includes a first processor core and a second processor core. On-chip memory is used to cache the fourth input tensor. The second direct memory access controller is used to read the fourth input tensor from on-chip memory and write the fourth input tensor to the first and second processor cores. The first processor core is used to perform the fifth operation corresponding to the fifth neural network layer in the neural network model based on the fourth input tensor. The second processor core is used to perform the sixth operation corresponding to the sixth neural network layer based on the fourth input tensor, in response to the fact that the sixth and fifth neural network layers in the neural network model duplicate the fourth input tensor.

[0006] A third embodiment of the present disclosure is a data processing method applied to a first processor core in a neural network processor, The first step is to cache the first input tensor corresponding to the first neural network layer in the neural network model using the first cache memory within the first processor core, The first direct memory access controller in the first processor core reads the second input tensor corresponding to the first neural network layer from the second cache memory, and writes the second input tensor to the arithmetic array in the first processor core. The present invention provides a data processing method that includes the steps of: reading a first input tensor from a first cache memory using an operational array; performing a first operation corresponding to a first neural network layer based on the first input tensor and the second input tensor; and obtaining a first output tensor.

[0007] A fourth embodiment of this disclosure is Processor and The processor includes memory for storing executable instructions, The processor is an electronic device used to realize the data processing method of the third embodiment described above by reading and executing executable instructions from memory.

[0008] An embodiment of a fifth aspect of this disclosure provides a computer-readable storage medium that stores a computer program for performing the data processing method of the third aspect described above. [Effects of the Invention]

[0009] In the neural network processor provided by the embodiments of this disclosure, the first processor core includes a first direct memory access controller, which can directly read the second input tensor stored in the second cache memory. This not only improves the read speed of the second input tensor but also reduces the amount of CPU bus bandwidth occupied and system power consumption by eliminating the need to frequently issue CPU instructions. Furthermore, the arithmetic array rapidly performs first operations corresponding to the first neural network layer on the first input tensor transmitted from the first cache memory and the second input tensor written from the first direct memory access controller, thereby improving the computational efficiency of the neural network processor in processing the neural network model. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of a single processor core provided by one exemplary embodiment of the present disclosure. [Figure 2] This is a schematic diagram of a neural network processor provided by one exemplary embodiment of the present disclosure. [Figure 3] This is a schematic diagram of another neural network processor provided by one exemplary embodiment of the present disclosure. [Figure 4] This is a schematic diagram of yet another neural network processor provided by one exemplary embodiment of the present disclosure. [Figure 5] This is a schematic diagram of yet another neural network processor provided by one exemplary embodiment of the present disclosure. [Figure 6A] This is a schematic diagram of yet another neural network processor provided by one exemplary embodiment of the present disclosure. [Figure 6B] This is a schematic diagram of yet another neural network processor provided by one exemplary embodiment of the present disclosure. [Figure 7]It is a schematic structural diagram of yet another neural network processor provided by one exemplary embodiment of the present disclosure. [Figure 8] It is a schematic structural diagram of a system-on-chip provided by one exemplary embodiment of the present disclosure. [Figure 9] It is a schematic structural diagram of another system-on-chip provided by one exemplary embodiment of the present disclosure. [Figure 10] It is a flowchart of a data processing method provided by one exemplary embodiment of the present disclosure. [Figure 11] It is a flowchart of another data processing method provided by one exemplary embodiment of the present disclosure. [Figure 12] It is a flowchart of yet another data processing method provided by one exemplary embodiment of the present disclosure. <00…… [Figure 13] It is a flowchart of yet another data processing method provided by one exemplary embodiment of the present disclosure. [Figure 14] It is a flowchart of yet another data processing method provided by one exemplary embodiment of the present disclosure. [Figure 15] It is a schematic structural diagram of an electronic device provided by one exemplary embodiment of the present disclosure.

Modes for Carrying Out the Invention

[0011] For the purpose of explaining the present disclosure, hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. Clearly, the described embodiments are only some embodiments of the present disclosure, not all embodiments. It should be understood that the present disclosure is not limited to the exemplary embodiments.

[0012] It should be noted that the relative arrangements, mathematical formulas, and numerical values of the components and steps described in these embodiments do not limit the scope of the present disclosure unless specifically described otherwise.

[0013] Summary of the Application Currently, the processor cores in the SoC's neural network processor primarily process at least one group of neural network layers in the neural network model, fully utilizing the SoC's computing resources to accelerate the neural network model's inference process and improve computational efficiency.

[0014] However, with the advancement of deep learning technology, the complexity of neural network models is increasing, and the number of neural network layers in a group of neural network layers in a neural network model is increasing. This, in turn, leads to an increasing amount of data for the weight parameters and / or data being processed that correspond to the neural network layers. If the amount of data for the weight parameters and / or data being processed that corresponds to a group of neural network layers exceeds the amount of data that can be stored in the L1M primary cache within the processor core, some of the weight parameters and / or data being processed must be stored in lower-level cache memory within the processor core or in memory outside the processor core.

[0015] In the following embodiment, we will illustrate with an example the case where the amount of data that can be stored in the primary cache L1M within the processor core is smaller than the amount of data of all the weight parameters and / or data to be processed corresponding to the group of neural network layers, and some of the weight parameters and / or data to be processed are stored in a lower-level secondary cache L2M within the processor core.

[0016] Figure 1 is a schematic diagram of a single processor core provided in one exemplary embodiment of the present disclosure. As shown in Figure 1, the processor core 10 may include a primary cache (L1M) 101, a systolic array 102, and a secondary cache (L2M) 103.

[0017] Here, communication between L1M101 and Systolic Array 102 can be conducted via an internal bus within the processor core 10, and communication between L1M101 and L2M can be conducted via the CPU bus.

[0018] L1M101 is used to cache some weight parameters and / or some data to be processed corresponding to at least one group of neural network layers that the processor core 10 needs to process. L2M103 is used to cache another set of weight parameters and / or other data to be processed corresponding to at least one group of neural network layers that the processor core 10 needs to process. Systolic array 102 is used to read the weight parameters and / or data to be processed in L1M101 based on an internal bus between systolic array 102 and L1M101, and to perform the corresponding operations on the weight parameters and / or data to be processed corresponding to at least one group of neural network layers.

[0019] As shown in Figure 1, let's take an example where the group of at least one neural network layer that the processor core 10 needs to process includes a first group of neural network layers, L1M101 stores some weight parameters corresponding to the first group of neural network layers, and L2M103 stores input feature data and other weight parameters corresponding to the first group of neural network layers. When the processor core 10 processes the first neural network layer among multiple neural network layers in the first group of neural network layers, it can read the input feature data stored in L2M103 using a CPU instruction, write the input feature data stored in L2M103 to L1M101, and perform the corresponding operation on the input feature data. When the processor core 10 processes other neural network layers among multiple neural network layers, if the weight data corresponding to those other neural network layers is stored in L2M103, it is necessary to read the weight parameters corresponding to those other neural network layers from L2M103 using a CPU instruction and perform the corresponding operation. In other words, when the processor core 10 processes the group of first neural network layers, it is necessary to issue CPU instructions frequently, which frequently occupies CPU bus bandwidth, increasing not only the amount of CPU bus bandwidth occupied and system power consumption, but also reducing the computational efficiency of the neural network processor in processing the neural network model.

[0020] Figure 2 is a schematic diagram of a neural network processor provided in one exemplary embodiment of the present disclosure. As shown in Figure 2, the neural network processor 20 may include a plurality of processor cores 10 as shown in Figure 1.

[0021] As shown in Figure 2, when multiple processor cores 10 in the neural network processor 20 simultaneously process a group of neural network layers corresponding to each processor core 10, CPU instructions are issued more frequently, and bus bandwidth is occupied more frequently. This significantly increases bandwidth usage and system power consumption, severely impacting the computational efficiency of the neural network processor in processing the neural network model.

[0022] Based on the technical challenges described above, embodiments of this disclosure provide a neural network processor in which a direct memory access controller is installed within the processor core of the neural network processor, and the direct memory access controller can establish access paths to lower-level cache memory within the processor core or memory outside the processor core. This allows the processor core to quickly read weight parameters or data to be processed stored in lower-level cache memory within the processor core or memory outside the processor core based on the direct memory access controller when processing a group of neural network layers. Therefore, this disclosure does not require frequent issuance of CPU instructions, can reduce bandwidth usage and system power consumption, and improves the computational efficiency of the neural network processor when processing neural network models.

[0023] Example circuit Figure 3 is a schematic diagram of another neural network processor provided by one exemplary embodiment of the present disclosure. As shown in Figure 3, the neural network processor 30 includes a first processor core 31. The first processor core 31 includes a first cache memory 311, a first direct memory access controller 312, and an arithmetic array 313.

[0024] Here, the first cache memory 311 is used to cache the first input tensor corresponding to the first neural network layer in the neural network model, the first direct memory access controller 312 is used to read the second input tensor corresponding to the first neural network layer from the second cache memory and write the second input tensor to the operation array 313, and the operation array 313 is used to read the first input tensor from the first cache memory 311, perform the first operation corresponding to the first neural network layer based on the first input tensor and the second input tensor, and obtain the first output tensor.

[0025] Exemplary, the neural network processor 30 may include one or more processor cores. The embodiments of this disclosure do not limit the number of processor cores included in the neural network processor 30. The embodiments of this disclosure exemplify an example in which the neural network processor 30 includes N processor cores, where N is an integer of 1 or more. The first processor core 31 may be any of the N processor cores.

[0026] Exemplary, the access speed of the first cache memory 311 is greater than the access speed of the second cache memory. In some examples, the first cache memory 311 may be a primary cache L1M within the first processor core 31. The second cache memory may be a secondary cache L2M within the first processor core 31, or it may be an L2M or DDR outside the first processor core 31, and the embodiments of this disclosure do not limit the type of the second cache memory. The first processor core 31 may or may not include the second cache memory; when the second cache memory is a secondary cache L2M within the first processor core 31, the first processor core 31 includes the second cache memory. When the second cache memory is an L2M or DDR outside the first processor core 31, the first processor core 31 does not include the second cache memory. The following embodiments exemplify the case where the second cache memory is DDR outside the first processor core 31.

[0027] The first direct memory access controller 312 may be a remote direct memory access (RDMA) controller installed between the second cache memory and the arithmetic array 313. The arithmetic array 313 may be a circuit for performing operations corresponding to a group of neural network layers, and is also called a systolic array. When the second input tensor is cached in the second cache memory, the first direct memory access controller 312 can directly access the second cache memory to read the second input tensor from the second cache memory and write the second input tensor to the arithmetic array 313, thereby enabling the arithmetic array 313 to perform the corresponding first operation based on the second input tensor.

[0028] The first neural network layer may be any neural network layer among a group of one or more neural network layers processed by the first processor core 31. Exemplarily, the group of one or more neural network layers processed by the first processor core 31 may include a plurality of neural network layers connected sequentially in series. In some examples, according to the order of each neural network layer in the plurality of neural network layers, the plurality of neural network layers may include a first neural network layer, a plurality of intermediate neural network layers, and a last neural network layer. The first neural network layer may be any neural network layer among the plurality of neural network layers. The following embodiment exemplifies the case where the first neural network layer is the first neural network layer among the plurality of neural network layers.

[0029] For example, if the first neural network layer is the first neural network layer in a group of neural network layers, then the data to be processed corresponding to the first neural network layer is the data to be processed corresponding to the group of neural network layers. In some examples, the data to be processed may be the feature data to be processed. For example, the data to be processed corresponding to the first neural network layer may be the first input feature data.

[0030] Let us take the example that the data to be processed corresponding to the first neural network layer is the first input feature data. In some examples, the first input tensor may be the first weight parameter corresponding to the first neural network layer, and the second input tensor may be the first input feature data. In some other examples, the first input tensor may be the first input feature data, and the second input tensor may be the first weight parameter corresponding to the first neural network layer. Embodiments of this disclosure do not limit the data types of the first and second input tensors. The following embodiments exemplify the case where the first input tensor is the first weight parameter and the second input tensor is the first input feature data.

[0031] Exemplary, the first operation corresponding to the first neural network layer may include at least one of linear transformations, convolutions, and pooling operations. The embodiments of this disclosure do not limit the type of the first operation corresponding to the first neural network layer, and the following examples illustrate the case in which the first operation corresponding to the first neural network layer includes linear operations. For example, the first operation corresponding to the first neural network layer may include multiplication operations.

[0032] To understand this, the SoC can write weight data and processing data corresponding to the neural network model to DDR during the initialization phase, and then write the weight data in DDR to the L1M of the corresponding processor core. If the capacity of the L1M of a processor core is smaller than the amount of data for the weight parameters corresponding to that processor core, some of the weight parameters corresponding to the processor core can be written to L1M, while other weight parameters corresponding to the processor core can continue to be stored in DDR or written to a lower level cache memory (e.g., L2M).

[0033] Let's take the example where the first input tensor is the first weight parameter, and the amount of data in the first weight parameter is less than the capacity of the first cache memory 311. The SOC can write the first weight parameter to the first cache memory 311, thereby caching the first input tensor in the first cache memory 311.

[0034] Exemplary, the first direct memory access controller 312 reading a second input tensor corresponding to a first neural network layer from a second cache memory and writing the second input tensor to an operational array 313 may include the first direct memory access controller 312 receiving a first read address and a first write address transmitted from the operational array 313, reading a second input tensor corresponding to a first neural network layer from the second cache memory based on the first read address, and writing a second input tensor to a corresponding storage location in the operational array 313 based on the first write address.

[0035] For example, the first read address is an address in the second cache memory that caches the second input tensor, and the first read address can be generated by the arithmetic array 313. The first write address is an address in the arithmetic array 313 that caches the second input tensor, and the first write address can also be generated by the arithmetic array 313. In some examples, the arithmetic array 313 may include an address generator. The address generator can be configured to have a starting address based on a configuration instruction. Furthermore, the address generator can calculate the first read address and the first write address based on the starting address and transmit the first read address and the first write address to the first direct memory access controller 312.

[0036] The arithmetic array 313 reading the first input tensor from the first cache memory 311 may include the arithmetic array 313 generating a second read address and a second write address, transmitting the second read address and the second write address to the first cache memory 311, the first cache memory 311 reading the first input tensor based on the second read address, and writing the first input tensor to the corresponding storage location in the arithmetic array 313 based on the second write address.

[0037] For example, the second read address is the address in the first cache memory 311 that stores the first input tensor, and the second read address can be generated by the arithmetic array 313. The second write address is the address in the arithmetic array 313 that stores the first input tensor, and the second write address can also be generated by the arithmetic array 313. In the embodiments of this disclosure, a detailed explanation of the implementation method for generating the second read address and the second write address by the arithmetic array 313 is omitted here.

[0038] In some examples, the first cache memory 311 may include a plurality of first storage units and a first cache controller. The first cache controller can read a first input tensor from a plurality of first storage units based on a second read address and send the first input tensor to a corresponding storage location in the arithmetic array 313 based on a second write address.

[0039] For example, taking the first operation as a multiplication operation, the operation array 313 can perform a multiplication operation on the first input tensor and the second input tensor to obtain the first output tensor of the first neural network layer. The first output tensor of the first neural network layer is also called the first output feature data.

[0040] In the neural network processor provided by the embodiments of this disclosure, the first processor core includes a first direct memory access controller, which can directly read the second input tensor stored in the second cache memory. This is unaffected by the occupation of CPU bus resources and avoids the frequent issuance of CPU instructions, thereby improving the read speed of the second input tensor and reducing the amount of CPU bus bandwidth occupied and system power consumption by eliminating the need to frequently issue CPU instructions. Furthermore, the arithmetic array can quickly perform first operations corresponding to the first neural network layer on the first input tensor transmitted from the first cache memory and the second input tensor written from the first direct memory access controller, improving the computational efficiency of the neural network processor in processing the neural network model.

[0041] In some other embodiments of this disclosure, the first input tensor and the second input tensor corresponding to the first neural network layer can be stored in the first cache memory 311. Thus, the arithmetic array 313 can directly read the first input tensor and the second input tensor from the first cache memory 311, perform the first operation corresponding to the first neural network layer based on the first input tensor and the second input tensor, and obtain the first output tensor.

[0042] In several other embodiments of this disclosure, the first and second input tensors corresponding to the first neural network layer may be stored in a second cache memory. Thus, the operational array 313 can read the first and second input tensors from the second cache memory based on the first direct memory access controller 312, perform a first operation corresponding to the first neural network layer based on the first and second input tensors, and obtain a first output tensor.

[0043] In some embodiments of the present disclosure, at least two neural network layers in a group of neural network layers that the first processor core 31 needs to process can correspond to the same input tensor (e.g., weight parameters), that is, the at least two neural network layers can reuse the same input tensor. The number of such at least two neural network layers may be two, three, or four, and embodiments of the present disclosure do not limit the number of neural network layers included in such at least two neural network layers. The following embodiments exemplify an example in which the at least two neural network layers include a second neural network layer and a third neural network layer.

[0044] In some examples, the second and third neural network layers in a group of neural network layers that the first processor core 31 needs to process utilize the same weight parameter twice, and this weight parameter is the third input tensor. Continuing to refer to Figure 3, the first direct memory access controller 312 is further used to read the third input tensor from the second cache memory and write it to the operation array 313 in response to the second and third neural network layers in the neural network model utilizing the third input tensor twice. The operation array 313 is further used to perform a second operation corresponding to the second neural network layer and a third operation corresponding to the third neural network layer based on the third input tensor.

[0045] For example, the second neural network layer may be any of the multiple neural network layers that the first processor core 31 needs to process, and the second neural network layer may be identical to or different from the first neural network layer. The embodiments of this disclosure do not limit the relationship between the second neural network layer and the first neural network layer. The following embodiments illustrate an example where the second neural network layer is different from the first neural network layer.

[0046] The third neural network layer may be any neural network layer different from the second neural network layer among the multiple neural network layers that the first processor core 31 needs to process, and the third neural network layer may be the same as the first neural network layer or different from the first neural network layer. The embodiments of this disclosure do not limit the relationship between the third neural network layer and the first neural network layer. The following embodiments will illustrate the case where the third neural network layer is different from the first neural network layer.

[0047] In some examples, the third neural network layer may be the neural network layer following the second neural network layer, or it may be a neural network layer located after the second neural network layer and connected to the second neural network layer via other neural network layers. The following embodiment exemplifies the case where the third neural network layer is the neural network layer following the second neural network layer.

[0048] In some examples, the third input tensor may be a second weight parameter used redundantly by the second and third neural network layers. In some examples, the third input tensor may be cached in the first cache memory 311 or in the second cache memory. Embodiments of this disclosure do not limit the cache location of the third input tensor. The following embodiments exemplify the case where the third input tensor is cached in the second cache memory.

[0049] For example, the second operation corresponding to the second neural network layer and the third operation corresponding to the third neural network layer may be the same as or different from the first operation corresponding to the first neural network layer. The embodiments of this disclosure do not limit the specific implementation methods of the second operation corresponding to the second neural network layer and the third operation corresponding to the third neural network layer.

[0050] Let's take the example of the third input tensor being cached in the second cache memory. Referring to Figure 3, since the second neural network layer and the third neural network layer utilize the third input tensor redundantly, when the arithmetic array 313 processes the second neural network layer and / or when the arithmetic array 313 processes the third neural network layer, it generates a third read address and a third write address and transmits the third read address and third write address to the first direct memory access controller 312. As a result, when the first direct memory access controller 312 receives the third read address and third write address, it can determine that the third neural network layer and the second neural network layer utilize the third input tensor redundantly. Based on the third read address and third write address, the first direct memory access controller 312 can read the third input tensor and write the third input tensor to the arithmetic array 313.

[0051] In some examples, the third read address may correspond to an address in the second cache memory that stores the third input tensor. The third write address may correspond to an address in the arithmetic array 313 that stores the third input tensor. Similar to the implementation of the first read address and / or the second read address, the third read address can also be generated by the arithmetic array 313. The implementation of the embodiment of this disclosure in which the arithmetic array 313 generates the third address will not be described in detail here.

[0052] For example, the first direct memory access controller 312 can read the second weight parameter from the second cache memory based on the third read address and write the second weight parameter to the corresponding storage location in the arithmetic array 313 based on the third write address.

[0053] In some examples, the computation array 313 performing a second operation corresponding to the second neural network layer based on the second weight parameter may include the computation array 313 performing a multiplication operation on the second input feature data and the second weight parameter based on the second input feature data corresponding to the second neural network layer to obtain second output feature data.

[0054] Let's take the example that the third neural network layer is the neural network layer following the second neural network layer. After the computation array 313 performs the second operation corresponding to the second neural network layer based on the second weight parameters, the computation array 313 can continue to process the third neural network layer. Furthermore, when the computation array 313 processes the third neural network layer, when generating the third read address and the third write address, the computation array 313 can directly perform the third operation corresponding to the third neural network layer based on the second weight parameters corresponding to the third write address.

[0055] In some examples, the computation array 313 directly performing a third operation corresponding to the third neural network layer based on the second weight parameters within the computation array 313 may include the computation array 313 directly performing a multiplication operation on the second weight parameters within the computation array 313 and the second output feature data (corresponding to the third input feature data of the third neural network layer) to obtain the third output feature data.

[0056] Let's take the example of a third neural network layer that is located after the second neural network layer and is connected to the second neural network layer via other neural network layers. When the computation array 313 processes the third neural network layer, it can regenerate the third read address and the third write address and send them to the first direct memory access controller 312. Furthermore, the first direct memory access controller 312 reads the second weight parameters based on the third read address and the third write address and writes the second weight parameters to the computation array 313. As a result, the computation array 313 can perform the third operation corresponding to the third neural network layer based on the second weight parameters.

[0057] In the neural network processor provided by the embodiments of this disclosure, when the second neural network layer and the third neural network layer utilize the third input tensor redundantly, and the third input tensor is cached in the second cache memory, the first direct memory access controller can quickly read the third input tensor. As a result, the computation array can quickly perform the second operation corresponding to the second neural network layer based on the third input tensor, and quickly perform the third operation corresponding to the third neural network layer based on the third input tensor, thereby improving the computational efficiency of the computation array.

[0058] In some other examples, at least two neural network layers in a group of neural network layers processed by the first processor core 31 can reuse the same input feature data. For example, the fourth neural network layer and the neural network layer following the first neural network layer in the group of neural network layers processed by the first processor core 31 reuse the same input feature data, and this input feature data is the first output tensor.

[0059] Continuing to refer to Figure 3, the arithmetic array 313 is further used to write the first output tensor to the first cache memory 311 and / or to the second cache memory via the first direct memory access controller 312, based on the duplicate utilization control instruction, in response to the duplicate utilization control instruction instructing the fourth neural network layer in the neural network model to duplicate the first output tensor.

[0060] The arithmetic array 313 is further used to read the first output tensor from the first cache memory 311 and / or to read the first output tensor from the second cache memory via the first direct memory access controller 312, and to perform a fourth operation corresponding to the fourth neural network layer based on the first output tensor.

[0061] To make it easier to understand, during the design phase of a neural network model, the input and output feature data for each neural network layer can be determined, thereby determining whether or not there is overlapping feature data usage within a group of neural network layers. If it is determined that there is overlapping feature data usage within a group of neural network layers, overlapping control instructions can be generated to control the corresponding computation arrays within the processor core to perform feature overlapping.

[0062] For example, a duplicate usage control instruction may be a control instruction for controlling the saving and retrieval of duplicated feature data. A duplicate usage control instruction may include an identifier for the duplicated feature and an address for saving the duplicated feature. In some examples, if the duplicated feature is the first output tensor, the identifier for the duplicated feature may be the identifier for the first output tensor, and the address for saving the duplicated feature may be the fourth write address in the first cache memory for storing the first output tensor, or the fifth write address in the second cache memory for storing the first output tensor.

[0063] Exemplary, the fourth neural network layer may be a neural network layer located after the first neural network layer in the group of neural network layers processed by the first processor core 31, and connected to the first neural network layer via other neural network layers. In some examples, the fourth neural network layer may be identical to the second or third neural network layer, or it may be different from the second and / or third neural network layer. Embodiments of this disclosure do not limit the relationship between the fourth neural network layer and the second and third neural network layers. The following embodiments exemplify an example in which the fourth neural network layer differs from the second and third neural network layers.

[0064] The fourth operation corresponding to the fourth neural network layer may be the same as or different from the first operation corresponding to the first neural network layer. The embodiments of this disclosure do not limit the specific implementation method of the fourth operation corresponding to the fourth neural network layer.

[0065] Illustratively, continuing to refer to Figure 3, the arithmetic array 313 can decide to duplicate the first output tensor based on the identifier of the duplicated feature in the duplicate usage control instruction. Furthermore, the arithmetic array 313 performs a first operation corresponding to the first neural network layer based on the first and second input tensors to obtain the first output tensor, and then obtains an address for saving the first output tensor in the duplicate usage control instruction. If the address for saving the first output tensor is the fourth write address in the first cache memory 311, the first output tensor is written to the fourth write address in the first cache memory. If the address for saving the first output tensor is the fifth write address in the second cache memory, the fifth write address and the first output tensor are sent to the first direct memory access controller 312, and the first direct memory access controller 312 stores the first output feature data at the fifth write address in the second cache memory in response to the fifth write address and the first output tensor. Here, the fourth write address may be an address for caching the first output tensor in the first cache memory 311, and the fifth write address may be an address for caching the first output tensor in the second cache memory.

[0066] Furthermore, when the first processor core 31 processes the fourth neural network layer, the arithmetic array 313 can generate a fourth read address (corresponding to the fourth write address) and a sixth write address and send them to the first cache memory 311. The first cache memory 311 can read the first output tensor based on the fourth read address and write the first output tensor to the corresponding storage location in the arithmetic array 313 based on the sixth write address. Here, the fourth read address is the address in the first cache memory 311 for caching the first output tensor, and the sixth write address is the address in the arithmetic array 313 for caching the first output tensor.

[0067] In some examples, when the first processor core 31 processes the fourth neural network layer, the arithmetic array 313 can generate a fifth read address (corresponding to the fifth write address) and a seventh write address and send them to the first direct memory access controller 312. The first direct memory access controller 312 can read the first output tensor from the second cache memory based on the fifth read address and write the first output tensor to the corresponding storage location in the arithmetic array 313 based on the seventh write address. Here, the fifth read address is the address in the second cache memory where the first output tensor is cached, and the seventh write address is the address in the arithmetic array 313 where the first output tensor is cached.

[0068] Since the generation method by which the arithmetic array 313 generates the fourth read address, sixth write address, fifth read address, and seventh write address is similar to the implementation method by which the arithmetic array 313 generates the first read address and first write address, a detailed explanation of the embodiments of this disclosure is omitted here.

[0069] The implementation method in which the first cache memory 311 reads the first output tensor based on the fourth read address and writes the first output tensor to the corresponding storage location in the operational array 313 based on the sixth write address is similar to the implementation method in which the first cache memory 311 reads the first input tensor based on the second read address and writes the first input tensor to the corresponding storage location in the operational array 313 based on the second write address. Therefore, a detailed explanation of the embodiments of this disclosure is omitted here.

[0070] The computation array 313 performing a fourth operation corresponding to the fourth neural network layer based on the first output tensor may include the computation array 313 performing a multiplication operation on the first output tensor and the third weight parameters corresponding to the fourth neural network layer to obtain the fourth output feature corresponding to the fourth neural network layer.

[0071] In the neural network processor provided by the embodiments of this disclosure, when the fourth neural network layer duplicates the first output tensor, the computation array stores the first output tensor in a first cache memory or a second cache memory. When the first processor core processes the fourth neural network layer, the first cache memory can write the first output tensor to the computation array, or the first direct memory access controller can quickly read the first output tensor and write it back to the computation array. This allows for the rapid execution of the fourth operation corresponding to the fourth neural network layer based on the first output tensor, thereby improving the computational efficiency of the computation array.

[0072] In some embodiments of this disclosure, at least two processor cores within a neural network processor can process neural network layers having the same weight parameters. That is, the at least two processor cores can duplicate the same weight parameters.

[0073] Exemplary, the number of processor cores in at least two processor cores may be any value between 2 and N. In some examples, at least two processor cores may include two processor cores. In some other examples, at least two processor cores may include three processor cores. Embodiments of the present disclosure do not limit the number of processor cores in at least two processor cores. Embodiments of the present disclosure exemplify an example in which at least two processor cores include a first processor core and a second processor core.

[0074] For example, the first processor core and the second processor core may duplicate the same weight parameters. In some examples, at least one neural network layer processed by the first processor core and at least one neural network layer processed by the second processor core may duplicate the same weight parameters. For example, one neural network layer processed by the first processor core and one neural network layer processed by the second processor core may duplicate the same weight parameters. Again, for example, two neural network layers processed by the first processor core and one neural network layer processed by the second processor core may duplicate the same weight parameters. Embodiments of this disclosure exemplify the case in which one neural network layer processed by the first processor core and one neural network layer processed by the second processor core duplicate the same weight parameters.

[0075] When the fifth neural network layer processed by the first processor core and the sixth neural network layer processed by the second processor core reuse the same weight parameters, and these weight parameters are the fourth input tensor, the fourth input tensor needs to be cached in the first or second cache memory corresponding to the first processor core, and in the first or second cache memory corresponding to the second processor core, respectively. In this way, the area overhead of the first or second cache memory increases.

[0076] Based on the above technical challenges, an embodiment of the present disclosure provides a neural network processor that caches the fourth input tensor, which is used in overlapping manner by the fifth neural network layer processed by the first processor core and the sixth neural network layer processed by the second processor core, in on-chip memory outside the neural network processor, and then installs a second direct memory access controller between the on-chip memory and the neural network processor. When the first processor core processes the fifth neural network layer, the second direct memory access controller reads the fourth input tensor from the on-chip memory to enable the first processor core to process the fifth neural network layer. When the second processor core processes the sixth neural network layer, the second direct memory access controller reads the fourth input tensor from the on-chip memory to enable the second processor core to process the sixth neural network layer.

[0077] As shown in Figure 4, the neural network processor 30 further includes a second processor core 32, based on the embodiment shown in Figure 3.

[0078] Here, the first processor core 31 reads the fourth input tensor corresponding to the fifth neural network layer in the neural network model from the on-chip memory via the second direct memory access controller, and is used to perform the fifth operation corresponding to the fifth neural network layer based on the fourth input tensor.

[0079] The second processor core 32 is used to read the fourth input tensor from on-chip memory via the second direct memory access controller in response to the sixth and fifth neural network layers in the neural network model making duplicate use of the fourth input tensor, and to perform the sixth operation corresponding to the sixth neural network layer based on the fourth input tensor.

[0080] Exemplary, the fifth neural network layer may be any of the multiple neural network layers processed by the first processor core 31 whose input tensor includes the fourth input tensor. In some examples, the fifth neural network layer may be identical to any of the first, second, third, and fourth neural network layers, or it may be different from the first, second, third, and fourth neural network layers. Embodiments of this disclosure do not limit the relationship between the fifth neural network layer and the first, second, third, and fourth neural network layers. Embodiments of this disclosure will illustrate, as an example, the case in which the fifth neural network layer is different from the first, second, third, and fourth neural network layers.

[0081] For example, the sixth neural network layer may be any of the neural network layers processed by the second processor core 32 whose input tensor includes the fourth input tensor. Since the implementation method is similar to that of the fifth neural network layer, a detailed explanation of the embodiments of this disclosure is omitted here.

[0082] Accordingly, the fifth operation corresponding to the fifth neural network layer and the sixth operation corresponding to the sixth neural network layer may be the same as any of the first operation corresponding to the first neural network layer, the second operation corresponding to the second neural network layer, the third operation corresponding to the third neural network layer, and the fourth operation corresponding to the fourth neural network layer, or they may be different from the first operation corresponding to the first neural network layer, the second operation corresponding to the second neural network layer, the third operation corresponding to the third neural network layer, and the fourth operation corresponding to the fourth neural network layer. The embodiments of this disclosure do not limit the types of the fifth operation corresponding to the fifth neural network layer and the sixth operation corresponding to the sixth neural network layer.

[0083] For example, the fourth input tensor may be a fourth weight parameter used by both the fifth and sixth neural network layers. In some examples, the fourth weight parameter may be the same as any of the first, second, and third weight parameters, or it may be different from the first, second, and third weight parameters. The embodiments of this disclosure do not limit the specific implementation of the fourth weight parameter. The embodiments of this disclosure will explain, as an example, that the fourth weight parameter is different from the first, second, and third weight parameters.

[0084] The on-chip memory may be DDR located outside the neural network processor on the SOC (i.e., outside the first processor core 31 and the second processor core 32). The on-chip memory can be used to store a fourth input tensor that is used redundantly by the first processor core 31 and the second processor core 32. In some examples, the on-chip memory can be used to store a fourth weight parameter.

[0085] The second direct memory access controller may be an RDMA coupled between the on-chip memory and the neural network processor.

[0086] In some examples, when the first processor core 31 processes the fifth neural network layer, it can generate a sixth read address and an eighth write address and send them to the second direct memory access controller, which can read the fourth weight parameter from on-chip memory based on the sixth read address and write the fourth weight parameter to the corresponding storage location in the first processor core 31 based on the eighth write address. Here, the sixth read address may be an address in on-chip memory that stores the fourth input tensor. The eighth write address may be an address in the first processor core 31 for caching the fourth input tensor.

[0087] In some examples, the eighth write address may be an address in the second cache memory of the first processor core 31. Therefore, after writing the fourth input tensor to the second cache memory, the first direct memory access controller 312 must then read the fourth input tensor from the second cache memory and write it to the arithmetic array 313.

[0088] The implementation method, which involves writing the fourth input tensor to the second cache memory, reading the fourth input tensor from the second cache memory using the first direct memory access controller 312, and writing the fourth input tensor to the arithmetic array 313, can be described by referring to the relevant explanation of the embodiment shown in Figure 3 above; therefore, a detailed explanation of the embodiment of this disclosure is omitted here.

[0089] The implementation that performs the fifth operation corresponding to the fifth neural network layer based on the fourth input tensor is similar to the implementation that performs the first operation corresponding to the first neural network layer based on the first and second input tensors, and the second operation corresponding to the second neural network layer based on the third input tensor, and so on. Therefore, a detailed explanation of the embodiments of this disclosure is omitted here.

[0090] In some cases, if the read address generated when the first processor core 31 processes the fifth neural network layer is the same as the read address generated when the second processor core 32 processes the sixth neural network layer, and is the sixth read address, then it is decided that the sixth neural network layer and the fifth neural network layer will reuse the fourth input tensor.

[0091] For example, when the second processor core 32 processes the sixth neural network layer, it can generate a sixth read address and a ninth write address and send them to the second direct memory access controller, which then reads the fourth weight parameter from on-chip memory based on the sixth read address and writes the fourth weight parameter to the corresponding memory location in the second processor core 32 based on the ninth write address, where the ninth write address is the address in the second processor core 32 for caching the fourth input tensor.

[0092] In some examples, the ninth write address may be an address in the second cache memory of the second processor core 32. Similar to the first processor core 31, after writing the fourth input tensor to the second cache memory of the second processor core 32, the first direct memory access controller in the second processor core 32 must further write the fourth input tensor to the arithmetic array in the second processor core 32.

[0093] The implementation method for writing the fourth input tensor to the second cache memory of the second processor core 32 and then writing the fourth input tensor to the arithmetic array of the second processor core 32 by the first direct memory access controller within the second processor core 32 can be described by referring to the relevant explanation of the embodiment shown in Figure 3 above; therefore, a detailed explanation of the embodiment of this disclosure is omitted here.

[0094] The implementation that performs the sixth operation corresponding to the sixth neural network layer based on the fourth input tensor is similar to the implementation that performs the first operation corresponding to the first neural network layer based on the first and second input tensors, and the second operation corresponding to the second neural network layer based on the third input tensor, and so on. Therefore, a detailed explanation of the embodiments of this disclosure is omitted here.

[0095] The neural network processor provided by the embodiments of this disclosure can accelerate the processing speed of the first processor core by having the second direct memory access controller quickly read the fourth input tensor corresponding to the fifth neural network layer from on-chip memory when the first processor core is processing the fifth neural network layer, and then executing the fifth operation corresponding to the fifth neural network layer based on the fourth input tensor. When the second processor core is processing the sixth neural network layer, the second direct memory access controller can quickly read the fourth input tensor corresponding to the sixth neural network layer from on-chip memory, and then execute the sixth operation corresponding to the sixth neural network layer based on the fourth input tensor, thereby accelerating the processing speed of the second processor core. Furthermore, since the fourth input tensor only needs to be stored in on-chip memory and does not need to be stored in the first and second processor cores respectively, the area overhead of the cache memory can be reduced.

[0096] As shown in Figure 5, based on the embodiment shown in Figure 3, the arithmetic array 313 can include a controller 3131, a third cache memory 3132, and an arithmetic circuit 3133.

[0097] Here, the controller 3131 is used to read the first input tensor from the first cache memory 311, write the first input tensor to the third cache memory, and generate arithmetic control signals.

[0098] The first direct memory access controller 312 is used to read the second input tensor from the second cache memory and write the second input tensor to the third cache memory 3132.

[0099] The third cache memory 3132 is used to cache the first input tensor and the second input tensor.

[0100] The arithmetic circuit 3133 is used to read the first input tensor and the second input tensor from the third cache memory in response to the arithmetic control signal, perform the first operation based on the first input tensor and the second input tensor, and obtain the first output tensor.

[0101] Exemplary, the controller 3131 may be hardware logic circuitry specifically designed to meet the particular needs of the systolic array. The controller 3131 can be used to manage and coordinate the flow of data and the operation of processing units throughout the entire arithmetic array.

[0102] The third cache memory 3132 can be used to cache the data to be processed by the arithmetic array 313. The arithmetic array 313 can only read data that has been cached in the third cache memory 3132.

[0103] The arithmetic circuit 3133 is the basic unit that performs the actual computational tasks in the systolic array. In some examples, the arithmetic circuit 3133 may include multiple processing elements (PEs) arranged according to certain rules, each PE being interconnected and working together to complete complex computational tasks.

[0104] In some examples, the second write address may be an address in the third cache memory 3132 of the arithmetic array 313 for caching the first input tensor. The controller 3131 can obtain the second read address and the second write address and send them to the first cache memory 311. The first cache memory 311 reads the first input tensor based on the second read address and writes the first input tensor to the corresponding location in the third cache memory 3132 based on the second write address, where the second write address is an address in the third cache memory 3132 of the arithmetic array 313.

[0105] In some examples, the first write address may be an address in the third cache memory 3132 of the arithmetic array 313 for caching the second input tensor. The first direct memory access controller 312 can write the second input tensor to the corresponding location in the third cache memory 3132 based on the first write address.

[0106] Exemplary, the arithmetic control signal may be a signal that controls the arithmetic circuit 3133 to perform a first arithmetic operation. In some examples, the arithmetic control signal may be used to control the start and end times of the operation, as well as the type of operation that the arithmetic circuit 3133 performs.

[0107] The neural network processor provided by the embodiments of this disclosure reads the first input tensor by the controller and writes the first input tensor to the third cache memory, and reads the second input tensor from the second cache memory by the first direct memory access controller and writes the second input tensor to the third cache memory. As a result, the arithmetic circuit performs a first operation corresponding to the operation control signal on the first and second input tensors, ensuring that each PE in the arithmetic circuit performs the computation task according to a preset order and rules, thereby achieving highly efficient parallel computing.

[0108] In some other embodiments, as shown in Figure 6A, the arithmetic array 313 may further include an address generator 3134 on the basis of the embodiment shown in Figure 5. The address generator 3134 may be coupled with a controller 3131. The address generator 3134 can generate a first read address, a first write address, a second read address, and a second write address, and can transmit the first read address, first write address, second read address, and second write address to the controller 3131. The controller 3131 transmits the first read address and first write address to the first direct memory access controller 312 and the second read address and second write address to the first cache memory 311.

[0109] In several other embodiments, as shown in Figure 6B, the arithmetic array 313 may further include an address generator 3134 on the basis of the embodiment shown in Figure 5. The address generator 3134 may be coupled with a first cache memory 311 and a first direct memory access controller 312. The address generator 3134 can generate a first read address, a first write address, a second read address, and a second write address, and can transmit the first read address and the first write address to the first direct memory access controller 312, and the second read address and the second write address to the first cache memory 311.

[0110] As a result, the first direct memory access controller 312 can read the second input tensor from the second cache memory based on the first read address and write the second input tensor to the third cache memory 3132 in the arithmetic array 313 based on the first write address, and the first cache memory 311 can read the first input tensor from the first cache memory 311 based on the second read address and write the first input tensor to the third cache memory 3132 in the arithmetic array 313 based on the second write address.

[0111] In some examples of this disclosure, as shown in Figure 7, the arithmetic array 313 may further include a selector 3135 on the basis of the embodiment shown in Figure 5.

[0112] Specifically, the controller 3131 is also used to generate a selection control signal, and the selector 3135 is used in response to the selection control signal to choose whether to output the first input tensor in the first cache memory 311 to the third cache memory, or to output the second input tensor in the first direct memory access controller 312 to the third cache memory 3132.

[0113] For example, the selector 3135 may be a binary selector and includes a first input terminal, a second input terminal, a control terminal, and an output terminal. The first input terminal may be connected to a first cache memory 311, the second input terminal may be connected to a first direct memory access controller 312, the control terminal may be connected to a controller 3131, and the output terminal may be connected to a third cache memory 3132.

[0114] Exemplary, the selection control signal may be a signal that controls the selector 3135 to form a first data transmission path between the first cache memory 311 and the third cache memory 3132, or a second data transmission path between the first direct memory access controller 312 and the third cache memory 3132. In some examples, the selection control signal includes a first voltage signal. In some other examples, the selection control signal includes a second voltage signal.

[0115] Exemplary, the first voltage signal and the second voltage signal may be signals with opposite levels. In some examples, the first voltage signal may be a signal corresponding to a logic high level "1", and the second voltage signal may be a signal corresponding to a logic low level "0". In some other examples, the first voltage signal may be a signal corresponding to a logic low level "1", and the second voltage signal may be a signal corresponding to a logic high level "0". Embodiments of this disclosure do not limit the implementation methods of the first voltage signal and the second voltage signal. Embodiments of this disclosure will be described as an example in which the first voltage signal is a signal corresponding to a logic high level "1" and the second voltage signal is a signal corresponding to a logic low level "0".

[0116] Let's take the case where the selection control signal is a first voltage signal as an example. In some examples, the controller 3131 can generate a first voltage signal when sending a second read address and a second write address to the first cache memory 311.

[0117] Let's take the case where the selection control signal is a second voltage signal as an example. In some examples, the controller 3131 can generate a second voltage signal when transmitting the first read address and the first write address to the first direct memory access controller 312.

[0118] Exemplary, in response to a selection control signal being a first voltage signal, selector 3125 can form a first data transmission path between a first cache memory 311 and a third cache memory 3132, thereby allowing the first cache memory 311 to read out a first input tensor and then transmit the first input tensor to the third cache memory 3132 based on the first data transmission path. Similarly, in response to a selection control signal being a second voltage signal, selector 3125 can form a second data transmission path between a first direct memory access controller 312 and a third cache memory 3132, thereby allowing the first direct memory access controller 312 to read out a second input tensor and then transmit the second input tensor to the third cache memory 3132 based on the second data transmission path.

[0119] The neural network processor provided by the embodiments of this disclosure includes a selector within the computation array, and a controller controls the selector to selectively form a first data transmission path between a first cache memory and a third cache memory, or a second data transmission path between a first direct memory access controller and a third cache memory. In this way, the first cache memory can write a first input tensor to the third cache memory based on the first data transmission path, and the first direct memory access controller can write a second input tensor to the third cache memory based on the second data transmission path, thereby enabling selection and switching of writing the first and second input tensors to the third cache memory.

[0120] In some embodiments of this disclosure, in order to save the area overhead of the cache memory, the compressed input tensor can be stored in the first cache memory 311 or the second cache memory. After the arithmetic array 313 reads the compressed input tensor, before the arithmetic circuit 3133 processes the input tensor, a decompression process is performed on the compressed input tensor to realize the corresponding operation on the input tensor. For example, the first input tensor cached in the first cache memory 311 may be the first input tensor after compression (compressed data), and / or the second input tensor cached in the second cache memory may be the second input tensor after compression. Embodiments of this disclosure will be described as an example where the first input tensor cached in the first cache memory 311 is the first input tensor after compression, and the second input tensor cached in the second cache memory is the second input tensor that has not been compressed.

[0121] In some cases, the first input tensor can be pre-compressed, the compressed first input tensor can be obtained, the compressed first input tensor can be written to the DDR (second cache memory) when the SOC is powered on, and the compressed first input tensor can be written from the DDR to the first cache memory 311, thereby caching the compressed first input tensor in the first cache memory 311.

[0122] In some other examples, the second input tensor can be pre-compressed, the compressed second input tensor can be obtained, and when the SOC is powered on, the compressed second input tensor can be written to the DDR (second cache memory), thereby caching the compressed second input tensor in the second cache memory.

[0123] In some embodiments of this disclosure, the first input tensor cached in the first cache memory 311 is the first input tensor after compression, and the second input tensor cached in the second cache memory is the second input tensor that has not been compressed. After the first cache memory 311 writes the first input tensor cached in the first cache memory 311 to the third cache memory 3132, referring to Figure 5, the controller 3131 further reads the first input tensor from the third cache memory 3132, decompresses the first input tensor in response to the first input tensor being compressed data, obtains the decompressed first input tensor, and writes the decompressed first input tensor to the third cache memory 3132. The third cache memory 3132 is further used to cache the decompressed first input tensor and second input tensor. The arithmetic circuit 3133, in response to the arithmetic control signal, reads the decompressed first input tensor and second input tensor from the third cache memory 3132, performs a first operation based on the decompressed first input tensor and second input tensor, and obtains the first output tensor.

[0124] For example, the controller 3131 can read the first input tensor from the third cache memory based on the read address corresponding to the second write address. If it determines that the read first input tensor is compressed data, it can call a decompression module (which may be located within or outside the arithmetic array 313, and the embodiments of this disclosure are not limited thereto) to decompress the first input tensor, obtain the decompressed first input tensor, and write the decompressed first input tensor to the third cache memory 3132 based on the second write address.

[0125] Exemplary, an implementation in which the arithmetic circuit 3133 responds to an arithmetic control signal to read the decompressed first input tensor and second input tensor from the third cache memory 3132, performs a first operation based on the decompressed first input tensor and second input tensor, and obtains a first output tensor is similar to an implementation in which the first input tensor and second input tensor are read from the third cache memory in response to an arithmetic control signal, perform a first operation based on the first input tensor and second input tensor, and obtain a first output tensor. Therefore, a detailed explanation of the embodiments of this disclosure is omitted here.

[0126] The neural network processor provided by the embodiments of this disclosure caches compressed data corresponding to a first input tensor in a first cache memory. When the first cache memory writes the compressed data to a third cache memory, the controller decompresses the compressed data, thereby caching the decompressed first input tensor in the third cache memory. The first operation is then performed based on the decompressed first input tensor. Since the first input tensor cached in the first cache memory is compressed data, the area overhead of the first cache memory can be reduced.

[0127] In some other embodiments of this disclosure, when the second neural network layer and the third neural network layer reuse the third input tensor, the third input tensor can be pre-compressed to save area overhead in the second cache memory, and the compressed third input tensor can be obtained. When the SOC is powered on, the compressed third input tensor is written to the second cache memory. Next, when the arithmetic array 313 processes the second neural network layer or the third neural network layer, the first direct memory access controller 312 can read the compressed third input tensor and write the compressed third input tensor to the third cache memory 3132 in the arithmetic array 313.

[0128] As a result, the controller 3131 is used to read the third input tensor from the third cache memory 3132, decompress the third input tensor in response to the fact that the third input tensor is compressed data, obtain the decompressed third input tensor, and write the decompressed third input tensor to the third cache memory 3132. The third cache memory 3132 is then used to cache the decompressed third input tensor. Taking the second neural network layer as an example, the arithmetic circuit 3133 is used to read the decompressed third input tensor from the third cache memory 3132 in response to the arithmetic control signal corresponding to the second neural network layer, and to perform the second operation based on the decompressed third input tensor.

[0129] In some further embodiments of the present disclosure, when the fourth neural network layer duplicates the first output tensor, in order to save area overhead in the first or second cache memory, the controller 3131 may first call a compression module (which may be located within or outside the arithmetic array 313, and the embodiments of the present disclosure are not limited thereto) to compress the first output tensor, obtain the compressed first output tensor, and then write the compressed first output tensor to the first cache memory 311 or the second cache memory.

[0130] As a result, the controller 3131 is used to read the first output tensor from the third cache memory 3132, decompress the first output tensor in response to the fact that the first output tensor is compressed data, obtain the decompressed first output tensor, and write the decompressed first output tensor to the third cache memory 3132, and the third cache memory 3132 is further used to cache the decompressed first output tensor. The arithmetic circuit 3133 is further used to read the decompressed first output tensor from the third cache memory 3132 in response to the arithmetic control signal corresponding to the fourth neural network layer, and to perform the fourth operation based on the decompressed first output tensor.

[0131] In several other embodiments of this disclosure, when the fifth neural network layer processed by the first processor core 31 and the sixth neural network layer processed by the second processor core 32 duplicate the fourth input tensor, the fourth input tensor can be pre-compressed to save on-chip memory area overhead, and the compressed fourth input tensor can be obtained. When the SOC is powered on, the compressed fourth input tensor is written to the on-chip memory.

[0132] Next, when the first processor core 31 processes the fifth neural network layer, the second direct memory access controller reads the compressed fourth input tensor from the on-chip memory and writes the compressed fourth input tensor to the second cache memory within the first processor core 31. Furthermore, the first direct memory access controller 312 within the first processor core 31 reads the compressed fourth input tensor from the second cache memory within the first processor core 31 and writes the compressed fourth input tensor to the third cache memory 3132 within the first processor core 31.

[0133] Simultaneously, when the second processor core 32 processes the sixth neural network layer, the second direct memory access controller can read the compressed fourth input tensor from the on-chip memory and write the compressed fourth input tensor to the second cache memory within the second processor core 32. Furthermore, the first direct memory access controller within the second processor core 32 can read the compressed fourth input tensor from the second cache memory within the second processor core 32 and write the compressed fourth input tensor to the third cache memory within the second processor core 32.

[0134] As a result, the controller 3131 in the first processor core 31 further reads the fourth input tensor from the third cache memory 3132, decompresses the fourth input tensor in response to the fact that the fourth input tensor is compressed data, obtains the decompressed fourth input tensor, and writes the decompressed fourth input tensor to the third cache memory 3132, which is then used to cache the decompressed fourth input tensor. The arithmetic circuit 3133 further reads the decompressed fourth input tensor from the third cache memory 3132 in response to the arithmetic control signal corresponding to the fifth neural network layer, and is used to perform the fifth operation based on the decompressed fourth input tensor.

[0135] Since the functions of the controller, the third cache memory, and the arithmetic circuits within the second processor core are similar to those of the controller, the third cache memory, and the arithmetic circuits within the first processor core, a detailed description of the embodiments of this disclosure is omitted here.

[0136] Building upon the above embodiments, embodiments of the present disclosure provide a system-on-a-chip. Figure 8 is a schematic diagram of a system-on-a-chip provided by one exemplary embodiment of the present disclosure. As shown in Figure 8, the system-on-a-chip 80 may include an on-chip memory 801, a second direct memory access controller 802, and a neural network processor 803, the neural network processor 803 including at least one processor core, the at least one processor core including a first processor core 8031 ​​and a second processor core 8032.

[0137] On-chip memory 801 is used to cache the fourth input tensor.

[0138] The second direct memory access controller 802 is used to read the fourth input tensor from the on-chip memory 801 and to write the fourth input tensor to the first processor core 8031 ​​and the second processor core 8032.

[0139] The first processor core, 8031, is used to perform the fifth operation, which corresponds to the fifth neural network layer in the neural network model, based on the fourth input tensor.

[0140] The second processor core 8032 is used to perform the sixth operation corresponding to the sixth neural network layer based on the fourth input tensor, in response to the fact that the sixth and fifth neural network layers in the neural network model duplicate the fourth input tensor.

[0141] As shown in Figure 9, on the basis of the embodiment shown in Figure 8, the first processor core 8031 ​​includes a second cache memory 901, a first direct memory access controller 902, and an arithmetic array 903.

[0142] The second direct memory access controller 802 is used to write the fourth input tensor to the second cache memory 901.

[0143] The second cache memory 901 is used to cache the fourth input tensor.

[0144] The first direct memory access controller 902 is used to read the fourth input tensor from the second cache memory 901 and write the fourth input tensor to the arithmetic array 903.

[0145] The arithmetic array 903 is used to perform a fifth operation based on the fourth input tensor.

[0146] Since the configuration of the second processor core 8032 is the same as that of the first processor core 8031, the embodiments of this disclosure omit a detailed description of the configuration of the second processor core 8032.

[0147] Building upon the above embodiments, embodiments of the present disclosure provide a data processing method applicable to a first processor core 31 in a neural network processor 30 shown in Figure 3. Figure 10 is a flowchart of a data processing method provided by one exemplary embodiment of the present disclosure. The data processing method includes the following steps 1001 to 1003.

[0148] In step 1001, the first input tensor corresponding to the first neural network layer in the neural network model is cached in the first cache memory within the first processor core.

[0149] In step 1002, the first direct memory access controller within the first processor core reads the second input tensor corresponding to the first neural network layer from the second cache memory and writes the second input tensor to the operational array.

[0150] In step 1003, the arithmetic array reads the first input tensor from the first cache memory and performs the first operation corresponding to the first neural network layer based on the first and second input tensors to obtain the first output tensor.

[0151] As shown in Figure 11, on the basis of the embodiment shown in Figure 10, the data processing method further includes the following steps 1004 and 1005.

[0152] In step 1004, the first direct memory access controller reads the third input tensor from the second cache memory and writes it to the operational array in response to the second and third neural network layers in the neural network model using the third input tensor redundantly.

[0153] In step 1005, the computation array performs a second operation corresponding to the second neural network layer based on the third input tensor, and also performs a third operation corresponding to the third neural network layer.

[0154] As shown in Figure 12, the data processing method, based on the embodiment shown in Figure 10, further includes the following steps 1006 and 1007.

[0155] In step 1006, the arithmetic array writes the first output tensor to the first cache memory and / or the first direct memory access controller writes the first output tensor to the second cache memory based on the duplicate use control instruction, in response to the duplicate use control instruction instructing the fourth neural network layer in the neural network model to duplicate use of the first output tensor.

[0156] In step 1007, the arithmetic array reads the first output tensor from the first cache memory and / or the first direct memory access controller reads the first output tensor from the second cache memory, and performs the fourth operation corresponding to the fourth neural network layer based on the first output tensor.

[0157] As shown in Figure 13, the data processing method, based on the embodiment shown in Figure 10, further includes the following steps 1008 and 1009.

[0158] In step 1008, the second direct memory access controller reads the fourth input tensor corresponding to the fifth neural network layer in the neural network model from the on-chip memory and performs the fifth operation corresponding to the fifth neural network layer based on the fourth input tensor.

[0159] In step 1009, the second processor core responds that the sixth and fifth neural network layers in the neural network model duplicate the fourth input tensor, and the second direct memory access controller reads the fourth input tensor from on-chip memory and performs the sixth operation corresponding to the sixth neural network layer based on the fourth input tensor.

[0160] As shown in Figure 14, based on the embodiment shown in Figure 10, step 1002 may include the following step 1401, and step 1003 may include the following steps 1402 to 1404.

[0161] In step 1401, the first direct memory access controller reads the second input tensor from the second cache memory and writes the second input tensor to the third cache memory.

[0162] In step 1402, the controller reads the first input tensor from the first cache memory, writes the first input tensor to the third cache memory, and generates an operation control signal.

[0163] In step 1403, the first and second input tensors are cached in the third cache memory.

[0164] In step 1404, the arithmetic circuit reads the first input tensor and the second input tensor from the third cache memory in response to the arithmetic control signal, performs the first operation based on the first input tensor and the second input tensor, and obtains the first output tensor.

[0165] In some embodiments of the present disclosure, the data processing method further includes generating a selection control signal by a controller. Writing a first input tensor to a third cache memory may include the selector choosing, in response to the selection control signal, to output the first input tensor in the first cache memory to the third cache memory. Writing a second input tensor to a third cache memory by a first direct memory access controller may include the selector choosing, in response to the selection control signal, to output the second input tensor in the first direct memory access controller to the third cache memory.

[0166] In some embodiments of the present disclosure, after the first cache memory has written the first input tensor to the third cache memory, the data processing method further includes the controller reading the first input tensor from the third cache memory, performing a decompression operation on the first input tensor in response to the first input tensor being compressed data, obtaining the decompressed first input tensor, and writing the decompressed first input tensor to the third cache memory. Accordingly, step 1404 specifically includes the arithmetic circuit reading the decompressed first input tensor and second input tensor from the third cache memory in response to an arithmetic control signal, performing a first operation based on the decompressed first input tensor and second input tensor, and obtaining a first output tensor.

[0167] Exemplary electronic device Figure 15 is a schematic diagram of an electronic device provided by one exemplary embodiment of the present disclosure. As shown in Figure 15, the electronic device 150 includes one or more processors 1501 and memory 1502.

[0168] The processor 1501 may be a central processing unit (CPU) or another form of processing unit having data processing capability and / or instruction execution capability, and can control other components in the electronic device 150 to perform desired functions.

[0169] Memory 1502 may include one or more computer program products, including various forms of computer-readable storage media such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or high-speed cache memory (cache). Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions can be stored in the computer-readable storage media, and the processor 1501 can execute the computer program instructions to realize the data processing methods and / or other desired functions of each embodiment of the present disclosure described above.

[0170] In one example, the electronic device 150 may further include an input device 1503 and an output device 1504, and these components are connected to each other via a bus system and / or other forms of connection mechanisms (not shown).

[0171] Naturally, for the sake of simplification, Figure 15 shows only some of the components of the electronic device 150 relevant to this disclosure, and components such as buses and input / output interfaces are omitted. Beyond this, the electronic device 150 may include any other appropriate components depending on the specific application.

[0172] Exemplary computer program products and computer-readable storage media The embodiments of this disclosure may also be computer program products that, in addition to the methods and apparatus described above, include computer program instructions that, when executed by a processor, cause the processor to perform steps in the data processing methods of the various embodiments of this disclosure described herein.

[0173] Computer program products can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java® and C++®, and common procedural programming languages ​​such as the C language or similar programming languages, to create program code for performing the operations of the embodiments of this disclosure. The program code may run entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server.

[0174] Furthermore, embodiments of the present disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform steps in the data processing methods of the various embodiments of the present disclosure described herein.

[0175] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0176] While the basic principles of this disclosure have been explained above with reference to specific examples, the advantages, advantages, and effects mentioned in this disclosure are merely illustrative and not limiting, and it is not assumed that these advantages, advantages, and effects must be present in each example of this disclosure. Furthermore, the specific details disclosed above are merely illustrative and intended to facilitate understanding, and are not limiting, nor do they imply that this disclosure must be implemented in the specific details described above.

[0177] Those skilled in the art can make various modifications and variations to this disclosure without departing from the spirit and scope of the application. Thus, if such modifications and variations of the application fall within the scope of the claims of this disclosure and the equivalent art, this disclosure is intended to include such modifications and variations.

Claims

1. A first cache memory used to cache the first input tensor corresponding to the first neural network layer in a neural network model, A first direct memory access controller used to read a second input tensor corresponding to the first neural network layer from a second cache memory and write the second input tensor to an operational array, A neural network processor including a first processor core, which includes an operation array used to read the first input tensor from the first cache memory, and to perform a first operation corresponding to the first neural network layer based on the first input tensor and the second input tensor, and to obtain a first output tensor.

2. The first direct memory access controller is further used to read the third input tensor from the second cache memory and write the third input tensor to the operational array in response to the second and third neural network layers in the neural network model making duplicate use of the third input tensor. The neural network processor according to claim 1, wherein the computation array is further used to perform a second operation corresponding to the second neural network layer based on the third input tensor, and to perform a third operation corresponding to the third neural network layer.

3. The arithmetic array is further used to write the first output tensor to the first cache memory and / or to the second cache memory by the first direct memory access controller in response to the duplicate use control instruction instructing the fourth neural network layer in the neural network model to duplicate use of the first output tensor. The neural network processor according to claim 1, wherein the arithmetic array is further used to read the first output tensor from the first cache memory and / or to read the first output tensor from the second cache memory by the first direct memory access controller, and to perform a fourth operation corresponding to the fourth neural network layer based on the first output tensor.

4. The neural network processor further includes a second processor core, The first processor core is used to read a fourth input tensor corresponding to the fifth neural network layer in the neural network model from the on-chip memory using a second direct memory access controller, and to perform a fifth operation corresponding to the fifth neural network layer based on the fourth input tensor. The neural network processor according to claim 1, wherein the second processor core is used to read the fourth input tensor from the on-chip memory by the second direct memory access controller in response to the sixth neural network layer and the fifth neural network layer in the neural network model making duplicate use of the fourth input tensor, and to perform a sixth operation corresponding to the sixth neural network layer based on the fourth input tensor.

5. The aforementioned arithmetic array includes a controller, a third cache memory, and an arithmetic circuit. The controller is used to read the first input tensor from the first cache memory, write the first input tensor to the third cache memory, and generate an operation control signal. The first direct memory access controller is used to read the second input tensor from the second cache memory and write the second input tensor to the third cache memory. The third cache memory is used to cache the first input tensor and the second input tensor. The neural network processor according to any one of claims 1 to 4, wherein the arithmetic circuit is used to read the first input tensor and the second input tensor from the third cache memory in response to the arithmetic control signal, to perform the first operation based on the first input tensor and the second input tensor, and to obtain the first output tensor.

6. The aforementioned arithmetic array further includes a selector, The controller is further used to generate selection control signals. The neural network processor according to claim 5, wherein the selector is used in response to the selection control signal to select whether to output the first input tensor in the first cache memory to the third cache memory or to output the second input tensor in the first direct memory access controller to the third cache memory.

7. The controller is further used to read the first input tensor from the third cache memory, decompress the first input tensor in response to the first input tensor being compressed data, obtain the decompressed first input tensor, and write the decompressed first input tensor to the third cache memory. The third cache memory is further used to cache the first input tensor and the second input tensor after the decompression process. The neural network processor according to claim 5, wherein the arithmetic circuit is used to read the first input tensor and the second input tensor after the decompression process from the third cache memory in response to the arithmetic control signal, to perform the first operation based on the first input tensor and the second input tensor after the decompression process, and to obtain the first output tensor.

8. The neural network processor according to any one of claims 1 to 4, wherein the first processor core includes the second cache memory.

9. The system includes on-chip memory, a second direct memory access controller, and a neural network processor, wherein the neural network processor includes at least one processor core, and the at least one processor core includes a first processor core and a second processor core. The aforementioned on-chip memory is used to cache the fourth input tensor. The second direct memory access controller is used to read the fourth input tensor from the on-chip memory and to write the fourth input tensor to the first processor core and the second processor core. The first processor core is used to perform a fifth operation corresponding to the fifth neural network layer in the neural network model based on the fourth input tensor. The second processor core is used to perform a sixth operation corresponding to the sixth neural network layer based on the fourth input tensor, in response to the sixth neural network layer and the fifth neural network layer in the neural network model making duplicate use of the fourth input tensor, in a system-on-chip.

10. The first processor core includes a second cache memory, a first direct memory access controller, and an arithmetic array. The second direct memory access controller is used to write the fourth input tensor to the second cache memory. The second cache memory is used to cache the fourth input tensor, The first direct memory access controller is used to read the fourth input tensor from the second cache memory and write the fourth input tensor to the arithmetic array. The system-on-chip according to claim 9, wherein the arithmetic array is used to perform the fifth operation based on the fourth input tensor.

11. A data processing method applied to a first processor core in a neural network processor, The steps include caching the first input tensor corresponding to the first neural network layer in the neural network model using the first cache memory within the first processor core, The first direct memory access controller in the first processor core reads the second input tensor corresponding to the first neural network layer from the second cache memory and writes the second input tensor to the arithmetic array in the first processor core, A data processing method comprising the steps of: reading the first input tensor from the first cache memory using the arithmetic array; performing a first operation corresponding to the first neural network layer based on the first input tensor and the second input tensor; and obtaining a first output tensor.

12. Processor and The processor includes a memory for storing executable instructions, The processor is an electronic device used to read and execute the executable instructions from the memory to realize the data processing method described in claim 11.

13. A computer-readable storage medium that stores a computer program for performing the data processing method described in claim 11 when executed by a processor.

Images