Data processing apparatus and neural network processor

Abstract

A data processing device and a neural network processor, the data processing device is applied to the neural network processor, the neural network processor further includes a memory module and a vector operation unit, the data processing device includes a first buffer, a second buffer, a storage conversion module and an instruction emitting module, the instruction emitting module is used to generate a first carrying instruction and a second carrying instruction;The storage conversion module is used to carry the relevant data of vector operation from the memory module to the first buffer according to the first carrying instruction, and carry the relevant data of vector operation from the first buffer to the second buffer according to the second carrying instruction.According to the above-mentioned data processing device, when the vector operation unit is operated, the number of memory access to the memory module can be reduced, so that the calculation efficiency is improved, and the overall performance of the neural network accelerator is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing device and a neural network processor. Background Technology

[0002] The "neural network," currently popular in the field of artificial intelligence, has penetrated various industries and become a common key component in various application areas. The flourishing development of deep neural networks has brought new challenges to the field of general-purpose computing. As researchers propose increasingly deeper neural network models, the computational demands of these models also increase. Existing general-purpose hardware (such as central processing units, CPUs) cannot meet the demands of large-scale and high-throughput computation, thus making it difficult to support the operation of large neural network models.

[0003] Therefore, existing technologies propose a neural network accelerator to meet the computational power requirements of neural networks during training and inference. It includes a memory module and a data processing module, and operates by transferring data (e.g., operands) stored in the memory module to a specific buffer in the data processing module, where the corresponding computation unit retrieves the data for computation. However, in existing neural network accelerators, the buffer capacity corresponding to the data used by the vector computation unit is relatively small. This results in a high number of memory accesses during vector computation, leading to reduced computational efficiency and hindering the improvement of the overall performance of the neural network accelerator. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data processing device and a neural network processor. According to the data processing device of this application embodiment, when the vector operation unit performs operations, the number of memory accesses to the memory module can be reduced, thereby improving the computational efficiency and thus improving the overall performance of the neural network accelerator.

[0005] In a first aspect, this application provides a data processing apparatus applied to a neural network processor. The neural network processor further includes a memory module and a vector operation unit. The data processing apparatus includes a first buffer, a second buffer, a storage conversion module, and an instruction issuing module. The memory module stores data related to vector operations. The instruction issuing module is used to generate a first transfer instruction and a second transfer instruction. The storage conversion module is used to transfer the data related to vector operations from the memory module to the first buffer according to the first transfer instruction, and to transfer the data related to vector operations from the first buffer to the second buffer according to the second transfer instruction. The capacity of the first buffer is larger than that of the second buffer. The vector operation unit obtains the data related to vector operations from the second buffer.

[0006] According to the data processing apparatus of this application embodiment, a first transfer instruction and a second transfer instruction are generated by an instruction issuing module. When data related to vector operations is transferred from the memory module to the second buffer, the storage conversion module, in response to the first transfer instruction, first transfers the data related to vector operations from the memory module to the first buffer for storage, and then, in response to the second transfer instruction, transfers the data related to vector operations from the first buffer to the second buffer for storage. This allows the vector operation unit to obtain the data related to vector operations from the second buffer. Since the capacity of the first buffer is larger than that of the second buffer, the amount of data transferred between the memory module and the data processing apparatus in one operation increases, reducing the number of memory accesses to the memory module, thereby improving vector operation efficiency and ultimately enhancing the overall performance of the neural network accelerator.

[0007] According to a first aspect, in a first possible implementation of the data processing device, generating the first transfer instruction and the second transfer instruction includes: recording attributes of the data related to the vector operation; and, upon receiving a transfer instruction from the memory module, generating the first transfer instruction based on the attributes, wherein the transfer instruction indicates that the data related to the vector operation be changed from a first storage level to a second storage level, the first storage level corresponding to the logical storage level of the memory module, and the second storage level corresponding to the logical storage level of the first buffer and the second buffer.

[0008] By generating transmission instructions that indicate the data transfer method at the logical storage level, data transfer instructions that indicate the data transfer method at the hardware level are generated. Since the logical storage level corresponds to the first buffer and the second buffer, compared with the existing technology where the logical storage level corresponds to the second buffer, the capacity of the logical storage level is expanded, allowing the program to use a larger logical storage level space.

[0009] According to the first possible implementation of the first aspect, in the second possible implementation of the data processing device, the attributes include at least one of the following: whether the data related to the vector operation is defined in the transmission instruction; the starting address and data size of the data related to the vector operation stored in the first buffer; the activity status of the data related to the vector operation; the dirty bit of the data related to the vector operation; the category of the data related to the vector operation; the usage frequency index of the data related to the vector operation; and the starting address, data size, and offset of the data related to the vector operation stored in the second buffer. Whether the data is defined in the transmission instruction indicates whether it is a declared variable in the transmission instruction. The activity status indicates the storage status in the first or second buffer. The dirty bit indicates whether the data stored in the first and second buffers are consistent. The category indicates whether the data is updated within a preset time period.

[0010] According to the second possible implementation of the first aspect, in the third possible implementation of the data processing device, generating the first transport instruction based on the attribute includes: generating the first transport instruction based on the starting address and data size of the relevant data of the vector operation stored in the first buffer.

[0011] Since the first data transfer instruction is derived from the starting address and size of the data stored in the first buffer, it can instruct the transfer of data with the same starting address and size stored in the first buffer, and the transfer method is from the memory module to the first buffer. This allows the data to be transferred to be determined based on the first data transfer instruction, achieving accurate data transfer.

[0012] According to any one of the first to third possible implementations of the first aspect, in a fourth possible implementation of the data processing device, the generation of the first transport instruction and the second transport instruction further includes: receiving a plurality of transport instructions from the memory module; for each of the plurality of transport instructions, a set of transport instructions between each transport instruction containing a clustering start identifier and the first subsequent transport instruction containing a clustering end identifier; determining the storage address in the second buffer of the data associated with the set of transport instructions in the relevant data of the vector operation, wherein the set of transport instructions is executed simultaneously; and generating the second transport instruction based on the attribute and the storage address.

[0013] For a set of transmission instructions that can be executed simultaneously, the instruction issuing module completes the storage address allocation of the data associated with the set of instructions in the second buffer. The second transfer instruction can be determined based on the storage address and data size of the data associated with the set of transmission instructions in the second buffer from the relevant data of vector operation, so that the data associated with the set of instructions can be transferred simultaneously, which can reduce the number of data transfers from the first buffer to the second buffer.

[0014] According to any one of the second to fourth possible implementations of the first aspect, in a fifth possible implementation of the data processing device, the instruction issuing module is further configured to: update at least one of the following attributes based on the current storage location of the relevant data for the vector operation: the starting address and data size of the relevant data for the vector operation stored in the first buffer, the activity state of the relevant data for the vector operation, and the starting address, data size, and offset of the relevant data for the vector operation stored in the second buffer.

[0015] This method ensures the real-time nature and accuracy of attributes, thereby guaranteeing the accuracy of subsequent data transfer.

[0016] According to any one of the first to fifth possible implementations of the first aspect, in a sixth possible implementation of the data processing device, the instruction issuing module is further configured to, when the remaining storage space of the second buffer is less than the data size of the data associated with the set of transmission instructions in the relevant data of the vector operation, generate a third transport instruction based on the attributes of the data stored in the second buffer, the third transport instruction instructing a portion of the data stored in the second buffer to be transported from the second buffer to the first buffer, wherein the sum of the data size of the portion of data and the remaining storage space of the second buffer is greater than or equal to the data size of the data associated with the set of transmission instructions in the relevant data of the vector operation; the storage conversion module is further configured to, according to the third transport instruction, transport the portion of data from the second buffer to the first buffer.

[0017] In this way, the second buffer has enough space to store the data associated with a set of transmission instructions in the relevant data of vector operations, so that the vector operation unit can perform data fetching and operation on the same set of transmission instructions at the same time, thereby minimizing the number of data transfers and improving data operation efficiency.

[0018] According to the sixth possible implementation of the first aspect, in the seventh possible implementation of the data processing device, the attributes of the data stored in the second buffer include categories, and the usage frequency of different categories of data is different. When the third transport instruction is generated, the usage frequency corresponding to the category of the partial data is less than or equal to the usage frequency corresponding to the category of the remaining data in the second buffer excluding the partial data.

[0019] Data is moved back to the first buffer by category. Since the category is associated with the frequency of use, the data moved back to the first buffer is the one with the lowest frequency of use, so as to save as many repeated data moves as possible.

[0020] According to any one of the fourth to seventh possible implementations of the first aspect, in the eighth possible implementation of the data processing device, the instruction issuing module is further configured to: segment each transmission instruction in the set of transmission instructions to obtain multiple instruction segments corresponding to each transmission instruction, wherein the instructions with the same segmentation order among the multiple instruction segments corresponding to each transmission instruction are combined to obtain multiple instruction sets, and the segmentation order of different instruction sets is different; execute the multiple instruction sets, wherein each instruction set in the multiple instruction sets is executed according to the segmentation order of the instructions, and each instruction in each instruction set is executed according to the order of its corresponding transmission instruction in the set of transmission instructions.

[0021] In this way, the number of times instructions are repeatedly executed can be reduced, and the temporal locality of the second buffer can be maintained.

[0022] According to any one of the second to eighth possible implementations of the first aspect, in the ninth possible implementation of the data processing device, the storage conversion module is further configured to periodically acquire the attributes of the vector operation-related data recorded by the instruction issuing module, and determine the change in the activity state of the vector operation-related data in the attributes.

[0023] This approach allows the command issuing module to only record the attributes of the data related to vector operations, reducing its data processing costs. Furthermore, it provides more options for triggering data transfer, making the data processing device more flexible in its operation.

[0024] According to any one of the fourth to ninth possible implementations of the first aspect, in the tenth possible implementation of the data processing device, the set of transmission instructions is obtained by clustering multiple transmission instructions. The clustering is completed based on the write or write information between data located in the second storage level and data located in the first storage level, as well as the dependency relationship between each data in the relevant data of the vector operation. The first transmission instruction in the set of transmission instructions obtained by clustering is given a clustering start identifier, and the last transmission instruction is given a clustering end identifier.

[0025] In this way, program segments with cluster start and cluster end identifiers can be recognized as a set of transmission instructions by the instruction issuing module of the data processing device, and processed according to the response operation to the set of transmission instructions.

[0026] According to the tenth possible implementation of the first aspect, in the eleventh possible implementation of the data processing device, the dependencies between the data related to the set of transmission instructions can form a directed acyclic graph.

[0027] By forming a directed acyclic graph of dependencies, a set of transfer instructions that can be executed simultaneously is determined, so that the data processing device can generate corresponding transfer instructions based on the set of transfer instructions and transfer the data involved in the transfer instructions.

[0028] According to any one of the second to eleventh possible implementations of the first aspect, in the twelfth possible implementation of the data processing device, the categories of the data related to vector operations located at the second storage level include a first category, a second category, and a third category. The usage frequency of the data in the second category is greater than that of the data in the third category, and the usage frequency of the data in the third category is greater than that of the data in the first category. The data in the first category is written by data other than itself in the data related to the vector operations within a first time threshold duration. The first time threshold duration is determined based on the total duration of the first threshold instructions, or the total duration of the first threshold time periods, or the total duration of the first threshold write or write-out operations. The data in the second category is written by itself within a second time threshold duration. The second time threshold duration is determined based on the total duration of the second threshold instructions, or the total duration of the second threshold time periods, or the total duration of the second threshold write or write-out operations. The data in the third category is not written by data other than itself in the data related to the vector operations within the first time threshold duration, and is not written by itself within the second time threshold duration.

[0029] In this way, the usage frequency of data related to vector operations located in the second storage level can be determined. When the remaining storage space in the second buffer is less than the data size of the data associated with a certain type of instruction within the data related to vector operations, a criterion is provided for the data processing device to determine the data to be moved back to the first buffer from the second buffer.

[0030] Secondly, this application provides a neural network processor, including a memory module and the data processing device described in any of the above.

[0031] Thirdly, this application provides a data processing method that can be executed by a compiler. The method includes: creating a second storage level, the second storage level corresponding to the logical storage level of the first buffer and the second buffer; setting the second storage level to be incompatible with the logical storage level corresponding only to the first buffer and the logical storage level corresponding only to the second buffer; determining the category of the data related to vector operations located in the second storage level; and clustering multiple transmission instructions.

[0032] Fourthly, this application provides an instruction acquisition method, the method comprising: defining a storage space parameter type, wherein the storage space parameter type includes a second storage level; and setting the second storage level to be incompatible with a logical storage level corresponding only to a first buffer and a logical storage level corresponding only to the second buffer; generating code according to the storage space parameter type, wherein the code is compiled to obtain a transmission instruction. Attached Figure Description

[0033] Figure 1 A schematic diagram illustrating an exemplary structure of a data processing module in a prior art neural network accelerator;

[0034] Figure 2 A schematic diagram illustrating the data access mechanism in the storage unit 101 of a prior art neural network accelerator;

[0035] Figure 3 This illustrates exemplary application scenarios of the data processing apparatus 30 according to embodiments of this application;

[0036] Figure 4 This diagram illustrates an exemplary structure of a data processing apparatus according to an embodiment of the present application.

[0037] Figure 5 An example of a dependency graph determined by transmission instructions ①②③ according to an embodiment of this application is shown. Detailed Implementation

[0038] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0039] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0040] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0041] The structure and function of existing neural network accelerators are described below. Existing neural network accelerators typically include a storage module and several data processing modules. The storage module and the data processing modules exchange data via a bus. The storage module is mainly used to store the data processed by the data processing modules and instructions indicating the data processing methods. The data processing modules are mainly used to perform scalar operations, vector operations, matrix operations, and some data transformation tasks according to the instructions.

[0042] Figure 1 An exemplary structural diagram of a data processing module in a prior art neural network accelerator is shown.

[0043] like Figure 1 As shown, the data processing module 10 includes a storage unit 101, a control unit 102, and an execution unit 103.

[0044] Storage unit 101 is a component that provides internal and external storage interaction functionality for data processing module 10. Data processing module 10 needs to load data from external storage module 20 into storage unit 101 within data processing module 10 in order to complete the corresponding data operations. Storage unit 101 includes several cache areas: L1 buffer, UB buffer, L0 buffer (including L0A buffer, L0B buffer, L0C buffer), general-purpose register (GPR), special-purpose register (SPR), etc.

[0045] The L1 buffer is a relatively large data transfer area within the data processing module 10, typically around 1MB in size. It can be used to temporarily store data that the data processing module 10 needs to repeatedly use during computation tasks, thereby reducing the number of data exchanges with the storage module 20 via the bus. The L1 buffer is also used to store data related to data conversion tasks, so that when the data processing module 10 performs some data conversion tasks, such as converting a 3D image to a 2D matrix (Img2Col), the unit performing the data conversion task (hereinafter referred to as the storage conversion unit) can retrieve data (operands) from the L1 buffer, complete the data conversion, and store the converted data in the L1 buffer.

[0046] The UB buffer is mainly used to store data related to vector operation tasks and scalar operation tasks, so that when the data processing module 10 executes some vector operation tasks or scalar operation tasks, the unit executing the vector operation task or scalar operation task (hereinafter referred to as the vector operation unit or scalar operation unit) can retrieve data (operands) from the UB buffer, complete the vector operation or scalar operation, and store the result of the operation in the UB buffer.

[0047] The L0 buffer, general-purpose registers, and special-purpose registers are used to store data related to matrix operation tasks, data related to scalar operation tasks, and data representing some computational behaviors of the data processing module, respectively.

[0048] The control unit 102 is responsible for coordinating the functions of multi-level instruction parallelism and synchronization control of dependent data within the data processing module 10. The control unit 102 includes: a system control module, a scalar instruction processing queue, an instruction issuing module, a matrix operation queue, a vector operation queue, a storage and conversion queue, etc.

[0049] The system control module is responsible for directing and coordinating the overall operation mode of the data processing module, configuring parameters, and controlling power consumption. The scalar instruction processing queue is used to decode instructions from the external storage module 20 and to store the decoded instructions indicating scalar operations. The decoded instructions indicating data transfer, vector operations, and matrix operations are sent to the storage conversion queue, vector operation queue, and matrix operation queue, respectively, via the instruction issuing module.

[0050] The execution unit 103 is a hardware unit that performs operations according to different instructions when the data processing module 10 actually executes tasks. It includes: a bus interface unit (BIU), a memory transfer element (MTE), a matrix operation unit, a vector operation unit, and a scalar operation unit. The bus interface unit is the interface between the data processing module 10 and the external bus, used to receive instructions and data from the external memory module 20. The memory transfer unit is used to transfer data between buffers and between the external memory module 20 and the buffers. The matrix operation unit, vector operation unit, and scalar operation unit are used to perform corresponding data calculation tasks based on the instructions stored in the matrix operation queue, vector operation queue, and scalar instruction processing queue, respectively. Specifically, the matrix operation unit performs matrix operations, the vector operation unit performs vector operations, and the scalar operation unit performs scalar calculations and program flow control.

[0051] The instructions from the external storage module 20 can be code written by a programmer or code automatically generated by operator code generation software and then compiled by a compiler. Before writing or automatically generating code, the corresponding logical storage levels are set according to the actual capacities of the L1 and UB buffers. When writing or automatically generating code, the code is written or generated according to the data transfer from one logical storage level to another. In the prior art, the compiler can create a logical storage level Global Memory corresponding to the storage module, a logical storage level Unified Buffer corresponding to the UB buffer, and a logical storage level L1 Buffer corresponding to the L1 buffer, so that the programmer can complete the program writing based on the data defined on the logical storage levels Global Memory, Unified Buffer, and L1 Buffer. In this case, the maximum usable capacity of the logical storage level L1 Buffer is 1MB, and the maximum usable capacity of the logical storage level Unified Buffer is 256KB. Therefore, when writing instructions for vector operations, programmers should ensure that the size of the data stored in the Unified Buffer at the logical storage level does not exceed 256KB.

[0052] Since the capacity of the logical storage level is the same as the actual hardware level L1 buffer and UB buffer, the storage conversion unit can directly perform data transfer operations from L1 buffer to UB buffer or from UB buffer to L1 buffer according to the decoded instructions.

[0053] As a component that stores the data used by the execution unit, the storage unit is closely related to the computing performance of the neural network accelerator. The rational design of the data access mechanism of the storage unit is a prerequisite for the entire accelerator to execute computing tasks efficiently. Figure 2 A schematic diagram illustrating the data access mechanism in the storage unit 101 of a prior art neural network accelerator is shown. Figure 2 As shown, the working method of the storage unit in the existing data processing module can be summarized in the following aspects.

[0054] Firstly, the data (operands) used by the matrix operation unit, vector operation unit, and scalar operation unit to perform operations according to the corresponding instructions (Cube instruction, Vector instruction, Scalar instruction) must be obtained from specific cache areas. Specifically, the operands of the matrix operation unit must be obtained from the L0 buffer, the operands of the vector operation unit must be obtained from at least one of the UB buffer or the L0C buffer, and the operands of the scalar operation unit must be obtained from at least one of the general-purpose registers, special-purpose registers, and the UB buffer.

[0055] Secondly, the data interaction between the storage module and each buffer, and between different buffers, needs to be handled by the storage conversion unit. For example, combining... Figure 1 When it is desired to transfer data from the L1 buffer to the L0A / L0B / UB buffer, a data transfer instruction MTE1 must be generated for the storage translation unit, so that the storage translation unit can transfer the data stored in the L1 buffer to the L0A / L0B / UB buffer according to the data transfer instruction MTE1. Similarly, when it is desired to transfer data from the storage module 20 to the L1 / L0B / UB buffer, a data transfer instruction MTE2 must be generated for the storage translation unit, and when it is desired to transfer data stored in the UB buffer to the storage module 20, a data transfer instruction MTE3 must be generated for the storage translation unit.

[0056] Thirdly, when there are no data dependencies involved in the instructions, instructions in different instruction processing queues can be executed simultaneously. A dependency means that a change in one piece of data (e.g., its value, type, etc.) will affect other data.

[0057] according to Figure 2As described above, existing neural network accelerators suffer from significant limitations and inflexibility in the data sources used by matrix operation units, vector operation units, and scalar operation units. Particularly for vector operation units, current technology only allows them to retrieve data from the UB buffer and L0C buffer. Due to the small capacity of the UB buffer (256KB), the data transfer process (storage module – UB buffer) for vector operations involves numerous accesses to the storage module, reducing the computational performance of the neural network accelerator. While the L1 buffer has a larger capacity (1MB), using it for vector operations (storage module – L1 buffer – UB buffer) complicates data partitioning strategies in actual programming, leading to an exponential increase in the data cost of exploring the optimal partitioning strategy and significantly increasing the computational cost of the neural network accelerator.

[0058] In view of this, embodiments of this application provide a data processing device and a neural network processor. According to the data processing device of this application embodiment, when the vector operation unit performs operations, the number of memory accesses to the memory module can be reduced, thereby improving the computational efficiency and thus improving the overall performance of the neural network accelerator.

[0059] Figure 3 This illustration shows an exemplary application scenario of the data processing apparatus 30 according to an embodiment of this application. For example, the data processing apparatus 30 can be applied to a neural network processor 1800, which can be mounted on terminal devices such as servers, edge computing terminals (terminals close to the data generation source), and vehicle-mounted devices. The neural network processor 1800 can operate in heterogeneous computing systems, such as... Figure 3 As shown, the first device can be a device with a central processing unit (CPU) as its core, used for processing routine tasks. The first device can distribute computing tasks to the second device. The second device can be a device responsible for executing computing tasks and its peripheral storage system. The device executing computing tasks in the second device can include at least one neural network processor 1800, and the peripheral storage system can be double datarate SDRAM (DDR) or high bandwidth memory (HBM), etc. At least one neural network processor 1800 can share the same peripheral storage system.

[0060] Figure 4 A schematic diagram of an exemplary structure of a data processing apparatus according to an embodiment of this application is shown.

[0061] In one possible implementation, such as Figure 4As shown, this application embodiment provides a data processing device 30 applied to a neural network processor 1800. The neural network processor 1800 further includes a memory module 40 and a vector operation unit 50. The data processing device 30 includes a first buffer 301, a second buffer 302, a storage conversion module 303, and an instruction issuing module 304. The memory module 40 stores data related to vector operations.

[0062] The instruction issuing module 304 is used to generate a first transport instruction and a second transport instruction;

[0063] The storage conversion module 303 is used to move the relevant data of vector operation from the memory module 40 to the first buffer 301 according to the first moving instruction, and to move the relevant data of vector operation from the first buffer 301 to the second buffer 302 according to the second moving instruction, wherein the capacity of the first buffer 301 is greater than that of the second buffer 302, and the vector operation unit 50 obtains the relevant data of vector operation from the second buffer 302.

[0064] Memory module 40 can be referred to above. Figure 1 And as shown in the related description, the storage module 20, the memory module 40 may be... Figure 4 And the related descriptions mention Double Rate Synchronous Dynamic Random Access Memory (DDR) or High Bandwidth Memory (HBM). The vector operation unit 50 can be referenced above. Figure 1 Examples of vector operation units in the related descriptions, and the first buffer 301 can be referred to above. Figure 1 And the example of the L1 buffer in the related description; the second buffer 302 can be referred to above. Figure 1 And examples of UB buffers in the related descriptions. Data related to vector operations, which may include operands for vector operations.

[0065] According to the data processing apparatus of this application embodiment, a first transfer instruction and a second transfer instruction are generated by an instruction issuing module. When data related to vector operations is transferred from the memory module to the second buffer, the storage conversion module, in response to the first transfer instruction, first transfers the data related to vector operations from the memory module to the first buffer for storage, and then, in response to the second transfer instruction, transfers the data related to vector operations from the first buffer to the second buffer for storage. This allows the vector operation unit to obtain the data related to vector operations from the second buffer. Since the capacity of the first buffer is larger than that of the second buffer, the amount of data transferred between the memory module and the data processing apparatus in one operation increases, reducing the number of memory accesses to the memory module, thereby improving vector operation efficiency and ultimately enhancing the overall performance of the neural network accelerator.

[0066] In addition to the functions of generating the first and second transport instructions described above, the instruction issuing module 303 of this application embodiment may also include the functions of a prior art instruction issuing module, so that the instruction issuing module 303 according to this application embodiment can either follow the working method of the prior art or operate according to the new working method of this application embodiment. In the new working method, the generated first and second transport instructions can be directly output to the storage conversion module, while other different types of instructions (vector type, matrix type, etc.) can be sent to the vector operation queue and matrix operation queue, etc., respectively. The following is an exemplary description of how the instruction issuing module 303 implements the function of generating the first and second transport instructions.

[0067] In one possible implementation, generating the first transport instruction and the second transport instruction includes:

[0068] Attributes that record data related to vector operations;

[0069] When a transmission instruction is received from memory module 40, a first transfer instruction is generated based on the attributes. The transmission instruction indicates that the data related to vector operations be changed from the first storage level to the second storage level. The first storage level corresponds to the logical storage level of memory module 40, and the second storage level corresponds to the logical storage level of the first buffer 301 and the second buffer 302.

[0070] By generating transmission instructions that indicate the data transfer method at the logical storage level, data transfer instructions that indicate the data transfer method at the hardware level are generated. Since the logical storage level corresponds to the first buffer and the second buffer, compared with the existing technology where the logical storage level corresponds to the second buffer, the capacity of the logical storage level is expanded, allowing the program to use a larger logical storage level space.

[0071] For example, the first storage level and the second storage level can be pre-defined logical storage levels, where the first storage level can correspond to memory module 40, and the second storage level can correspond to first buffer 301 and second buffer 302. Embodiments of this application allow programmers to write code or operator code generation software to automatically generate code, for data transfer from the memory module to the first / second buffer, according to the data being transferred from the first logical storage level to the second logical storage level. This part of the work can be completed on the first device. Since the capacity of the second storage level can be the sum of the capacities of the first and second buffers, the capacity of the second storage level is greater than the capacity of the logical storage level corresponding to the second / first buffer in the prior art, allowing for the use of a larger logical storage level space when writing programs.

[0072] On the first device, the code is compiled by a compiler to obtain transmission instructions, which are then distributed to the second device as computational tasks. On the second device, the transmission instructions can be stored in the memory module 40 and transmitted to a neural network processor 1800 via a bus. During program initialization, the instruction issuing module 304 of the data processing device 30 of the neural network processor 1800 can record and store the attributes of the data related to vector operations, for example, in the form of a table. Each piece of data can correspond to at least one attribute. Upon receiving a transmission instruction from the memory module 40, the instruction issuing module 304 can generate a first transfer instruction based on the transmission instruction. Therefore, the storage conversion module 303 can directly execute the data transfer operation from the memory module to the first buffer based on the first transfer instruction.

[0073] The following is an example of the attributes of data related to vector operations. In one possible implementation, the attributes include whether the data related to the vector operation is defined in the transmission instruction, the starting address and size of the data related to the vector operation stored in the first buffer, the activity status of the data related to the vector operation, the dirty bit of the data related to the vector operation, the category of the data related to the vector operation, the usage frequency index of the data related to the vector operation, and at least one of the starting address, size, and offset of the data related to the vector operation stored in the second buffer. Whether it is defined in the transmission instruction indicates whether it is a declared variable in the transmission instruction; the activity status indicates the storage status in the first or second buffer; the dirty bit indicates whether the data stored in the first and second buffers are consistent; and the category indicates whether the data has been updated within a preset time period.

[0074] The determination of whether the data related to vector operations is defined in the transmission instructions, indicating whether it is a declared variable, can be expressed in multiple ways. For example, "1" can be used to indicate that the data is defined, and "0" to indicate that the data is not defined. Alternatively, if the data is defined, the neural network accelerator will automatically allocate an address in an L1 buffer for that data; this address can also be used to indicate that the data is defined. If a data item does not have a corresponding address, it means that the data is not defined. This application does not restrict the way in which the data related to vector operations is defined in the transmission instructions.

[0075] The starting address and size of the data stored in the first buffer for vector operations indicate the storage location of the data in the first buffer. For example, a starting address of 0 and a data size of 256K indicate that the data is stored in the first buffer at positions 0 to 255. The starting address and size of the data stored in the first buffer have specific values ​​when the data is defined. For undefined data, the starting address and size of the data stored in the first buffer can be null.

[0076] The activity state of data related to vector operations indicates the storage state of the data in the first or second buffer. Data stored in the first buffer is considered "inactive" (represented by "0"), while data stored in the second buffer is considered "active" (represented by "1"). For example, in this embodiment, the data used by the vector operation unit to complete one operation can be moved from the memory module to the first buffer, and then from the first buffer to the second buffer. The data stored in the second buffer can be directly retrieved and calculated by the vector operation unit. The data in the first buffer requires a data transfer from the first buffer to the second buffer before it can be processed. Therefore, the data in the second buffer can be considered "ready" for operation, while the data in the first buffer is "not ready" for operation. If "activity" represents "readiness," then the data stored in the second buffer is "active" or has strong "activity" compared to the data stored in the first buffer, and can be represented by "1"; the data stored in the first buffer is "inactive" or has weak "activity," and can be represented by "0."

[0077] The dirty bits in vector operation-related data indicate whether the data stored in the first and second buffers are consistent. For example, if data is written to the second buffer, the data may change, causing corresponding changes in the data size and storage address in the second buffer, which may lead to inconsistencies between the first and second buffers for the same data. For instance, "1" can be used to indicate that the data stored in the first and second buffers are inconsistent, and "0" can be used to indicate that the data stored in the first and second buffers are consistent.

[0078] The category of data related to vector operations indicates the update status of the data within a preset time period. For example, data may be used repeatedly during vector operations, which may lead to updates of data that depend on it. For instance, in the vector addition C = A + B, vector data A, vector data B, and vector data C can be logically defined as different data at the second storage level. Vector data C can be written to by both vector data A and vector data B. Based on the above addition relationship, vector data C can be data that depends on both vector data A and vector data B. Updates to either vector data A or vector data B will cause vector data C to update accordingly. Based on the update status of data that depends on vector data C, the update status of vector data C within the preset time period can be determined, i.e., the category of vector data C can be determined. Update status can include being written to itself within the preset time period and being written to other data besides itself. Categories can be divided into different categories such as Category 1, Category 2, and Category 3 according to the update status. An exemplary method for determining the data category can be found in the relevant description in the compiler section below.

[0079] The frequency index of data related to vector operations can be used to represent the frequency of data usage determined by the Least Recently Used (LRU) algorithm. The design principle of the LRU algorithm is that if a piece of data has not been accessed recently, it is unlikely to be accessed in the future. In other words, when the limited space is full, the least recently accessed data should be discarded. Taking N data points (N > 1 and an integer) as an example, if the N data points determined by the LRU algorithm have M usage frequencies (1 < M ≤ N and an integer), then the frequency indices of the N data points, in order of frequency, can be represented by numbers from "1" to "M," with a larger index indicating a higher usage frequency.

[0080] The starting address, size, and offset of the data related to vector operations stored in the second buffer indicate the storage location of the data in the second buffer. For example, a starting address of 0 and a data size of 256KB indicate that the data is stored at address 0-255 in the second buffer. The offset indicates the deviation of the data's storage address. For example, when the offset is represented by "1", the storage location of the data related to vector operations in the second buffer can be from 1 to 256.

[0081] Those skilled in the art will understand that the attributes recorded and stored by the instruction issuing module are not limited to the examples described above. This application does not impose any limitations on the specific types and number of attributes.

[0082] The first and second transport instructions can be generated by combining attributes. Below is an example of how the first transport instruction is generated based on attributes.

[0083] In one possible implementation, generating the first transport instruction based on the attribute includes: generating the first transport instruction based on the starting address and data size of the relevant data stored in the first buffer for vector operations.

[0084] For example, as described above, the data defined in the transfer instruction corresponds to a starting address and data size allocated and stored in the first buffer. The first transfer instruction can instruct the transfer of data with the starting address and data size stored in the first buffer. In this case, the first transfer instruction can be generated based on the storage address and data size.

[0085] Since the first data transfer instruction is derived from the starting address and size of the data stored in the first buffer, it can instruct the transfer of data with the same starting address and size stored in the first buffer, and the transfer method is from the memory module to the first buffer. This allows the data to be transferred to be determined based on the first data transfer instruction, achieving accurate data transfer.

[0086] The following describes an example of the second transport instruction being generated based on attributes. In one possible implementation, generating the first and second transport instructions further includes: receiving multiple transport instructions from a memory module; for each set of transport instructions from a transport instruction containing a cluster start identifier to the first transport instruction containing a cluster end identifier thereafter; determining the storage address in a second buffer for data associated with a set of transport instructions in the vector operation-related data, wherein the set of transport instructions is executed simultaneously; and generating the second transport instruction based on the attributes and the storage address.

[0087] For a set of transmission instructions that can be executed simultaneously, the instruction issuing module completes the storage address allocation of the data associated with the set of instructions in the second buffer. The second transfer instruction can be determined based on the storage address and data size of the data associated with the set of transmission instructions in the second buffer from the relevant data of vector operation, so that the data associated with the set of instructions can be transferred simultaneously, which can reduce the number of data transfers from the first buffer to the second buffer.

[0088] For example, multiple transfer instructions from the memory module may include transfer instructions that have already been clustered into a group by the compiler. Since these clustered transfer instructions can be executed simultaneously, reducing the time required to move all the data used by that group of instructions to the second buffer can improve the computational efficiency of the vector operation unit.

[0089] Clustering can be performed by the compiler. One exemplary method for the instruction issuing module to determine a group of transmission instructions is to configure the compiler to add a cluster start identifier (e.g., ".group_st") to the first instruction that can be clustered into a group, and a cluster end identifier (e.g., ".group_ed") to the last instruction that can be clustered into a group. When receiving transmission instructions, if the instruction issuing module identifies an instruction containing the ".group_st" identifier, it stores that instruction in the instruction receiving module's instruction buffer and continues receiving instructions until it receives the first instruction containing the ".group_ed" identifier. In this case, all instructions in the instruction buffer from the instruction containing ".group_st" to the instruction containing ".group_ed" constitute a group of transmission instructions.

[0090] Those skilled in the art will understand that the cluster start identifier and cluster end identifier are not limited to the examples above. This application does not impose any restrictions on the specific design of the cluster start identifier and cluster end identifier.

[0091] Another exemplary method for the instruction dispatch module to determine a group of transport instructions is to configure the compiler to add the same cluster identifier (e.g., ".group_A") to each instruction that can be clustered into a group when clustering transport instructions, and to ensure that the cluster identifiers for different groups of transport instructions are different. When receiving transport instructions, if the instruction dispatch module identifies an instruction containing the identifier ".group_A", it stores the instruction in the instruction receive module's instruction buffer and continues to receive instructions until all instructions containing the identifier ".group_A" are received.

[0092] An exemplary implementation of the compiler adding identifiers to transfer instructions can be found in the relevant description in the compiler section below.

[0093] Those skilled in the art should understand that the method described above is not the only way for the instruction issuing module to determine a set of transmission instructions. This application does not limit the specific method used by the instruction issuing module to determine a set of transmission instructions.

[0094] Upon receiving a set of transmission instructions, the instruction issuing module can generate a second transfer instruction based on the attributes of the data associated with the set of transmission instructions in the relevant data of vector operations, such as the starting address and data size stored in the second buffer. The second transfer instruction can instruct the transfer of data with the starting address and data size stored in the second buffer, and the transfer method is from the first buffer to the second buffer.

[0095] In one possible implementation, the instruction issuing module is further configured to: update at least one of the following attributes based on the current storage location of the data related to the vector operation: the starting address and data size of the data related to the vector operation stored in the first buffer, the activity state of the data related to the vector operation, and the starting address, data size, and offset of the data related to the vector operation stored in the second buffer.

[0096] For example, after data related to vector operations undergoes data movement, its storage state (e.g., storage location) may change. This storage state is related to the starting address, data size, and activity level stored in the first buffer, as well as the starting address, data size, and offset stored in the second buffer. As described above, the first and second data movement instructions are generated based on the attributes. Therefore, after data movement, the attributes can be updated according to the current storage location of the data.

[0097] This method ensures the real-time nature and accuracy of attributes, thereby guaranteeing the accuracy of subsequent data transfer.

[0098] In one possible implementation, the instruction issuing module is further configured to, when the remaining storage space in the second buffer is less than the data size of the data associated with a set of transmission instructions in the relevant data of vector operations, generate a third transfer instruction based on the attributes of the data stored in the second buffer. The third transfer instruction instructs a portion of the data stored in the second buffer to be transferred from the second buffer to the first buffer, wherein the sum of the size of the portion of data and the remaining storage space in the second buffer is greater than or equal to the size of the data associated with a set of transmission instructions in the relevant data of vector operations. The storage conversion module is further configured to transfer the portion of data from the second buffer to the first buffer according to the third transfer instruction.

[0099] In this way, the second buffer has enough space to store the data associated with a set of transmission instructions in the relevant data of vector operations, so that the vector operation unit can perform data fetching and operation on the same set of transmission instructions at the same time, thereby minimizing the number of data transfers and improving data operation efficiency.

[0100] For example, when the instruction issuing module allocates storage addresses in the second buffer for a set of transmission instructions that can be executed simultaneously, the second buffer may be insufficient, meaning the remaining storage space is less than the size of the data associated with the transmission instructions in the vector operation data. In this case, the data associated with the transmission instructions may not be able to be moved to the second buffer all at once, increasing the number of data moves and reducing the computational efficiency of the neural network processor. Therefore, this embodiment proposes that the instruction issuing module can select a portion of the data that can be moved back to the first buffer from the second buffer based on attributes and preset rules, and generate a third transfer instruction based on the attributes of this portion of data. The storage conversion module executes the third transfer instruction to complete the data transfer from the second buffer to the first buffer, ensuring sufficient space in the second buffer.

[0101] The sum of the selected data size and the remaining storage space in the second buffer can be greater than or equal to the data size of the data associated with a set of transfer instructions in the relevant data of the vector operation. The third transfer instruction can be generated based on the storage address and size of this selected data in the second buffer. The third transfer instruction can instruct the transfer of this selected data from the second buffer to the first buffer.

[0102] Based on the third transfer instruction, the storage conversion module can accurately locate a portion of the data in the second buffer and transfer it to the first buffer.

[0103] The following describes an example of how to set up preset rules. One example of a preset rule is to determine the transfer priority based on the category of the data related to vector operations in the attributes.

[0104] In one possible implementation, the attributes of the data stored in the second buffer include categories. Different categories of data have different usage frequencies. When generating the third transport instruction, the usage frequency corresponding to the category of some data is less than or equal to the usage frequency corresponding to the category of the remaining data in the second buffer, excluding some data.

[0105] Data is moved back to the first buffer by category. Since the category is associated with the frequency of use, the data moved back to the first buffer is the one with the lowest frequency of use, so as to save as many repeated data moves as possible.

[0106] For example, data stored in the second buffer can be moved sequentially according to its category, in the order of no-category data, first-category data, third-category data, and second-category data. No-category data can be data in the second buffer that is unrelated to vector operations. First-category, third-category, and second-category data can be data in the second buffer that is related to vector operations. The category indicates whether the data is updated within a preset time period, and can include whether the data is written to itself or whether it is written to other data. The frequency of data usage may differ under different update conditions. In this embodiment, if data is written to other data within a first time threshold duration, the usage frequency is considered relatively low; if data is written to itself within a second time threshold duration, the usage frequency is considered relatively high; if data is neither written to itself within the first time threshold duration nor within the second time threshold duration, the usage frequency is considered moderate. Exemplary methods for selecting the first and second time thresholds and exemplary implementations for determining different categories can be found in the relevant description in the compiler section below. The usage frequency of data in the first category can be lower than that of data in the third category, and the usage frequency of data in the third category can be lower than that of data in the second category. For data in the same category, priority can be determined by ranking the usage frequency index of the attribute from low to high.

[0107] Those skilled in the art should understand that the data transfer priority in the preset rules can also be determined by other attributes, such as dirty bits. This application does not limit the specific implementation of the preset rules.

[0108] In one possible implementation, the instruction issuing module is further configured to: segment each transmission instruction in a set of transmission instructions to obtain multiple instruction segments corresponding to each transmission instruction; combine the instructions with the same segmentation order among the multiple instruction segments corresponding to each transmission instruction to obtain multiple instruction sets, wherein the segmentation order of different instruction sets is different; execute multiple instruction sets, wherein each instruction set in the multiple instruction sets is executed according to the segmentation order of the instructions, and each instruction in each instruction set is executed according to the order of its corresponding transmission instruction in a set of transmission instructions.

[0109] In this way, the number of times instructions are repeatedly executed can be reduced, and the temporal locality of the second buffer can be maintained.

[0110] For example, consider a set of transmission instructions including instructions ①, ②, and ③. Each instruction may need to be executed repeatedly. The frequency of instruction re-execution is related to the temporal locality of the second buffer. If an instruction is executed more frequently, it means that the vector processing unit retrieves the data associated with that instruction from the second buffer more often. Therefore, the temporal locality of the data associated with that instruction stored in the second buffer is relatively strong. Similarly, if an instruction is executed less frequently, it means that the vector processing unit retrieves the data associated with that instruction from the second buffer less often. Therefore, the temporal locality of the data associated with that instruction stored in the second buffer is relatively weak. If there is a large amount of data with weak temporal locality in the second buffer, the overall temporal locality of the second buffer will be relatively reduced, which is detrimental to improving data transfer efficiency.

[0111] In this case, transmission instructions ①, ②, and ③ can be segmented and combined to obtain multiple instruction sets. For example, transmission instructions ①, ②, and ③ can be segmented into two parts each, resulting in two transmission instructions ①-A and ①-B corresponding to transmission instruction ①, ②-A and ②-B corresponding to transmission instruction ②, and ③-A and ③-B corresponding to transmission instruction ③. Among the multiple instruction segments obtained after the above segmentation, ①-A, ②-A, and ③-A can be instructions with the same segmentation order, and ①-B, ②-B, and ③-B can be instructions with the same segmentation order. Instructions with the same segmentation order are combined to obtain two instruction sets A and B, with different segmentation orders for different instruction sets. In this case, instruction set A includes multiple instruction segments ①-A, ②-A, and ③-A, and instruction set B includes multiple instruction segments ①-B, ②-B, and ③-B.

[0112] When executing two instruction sets A and B, they are executed according to the order in which the instructions are segmented. In this example, instruction set A is executed first, followed by instruction set B. Each instruction within each instruction set is executed in the order of its corresponding transfer instruction within a group of transfer instructions. In this example, the multiple instruction segments ①-A, ②-A, and ③-A in instruction set A are executed in the order of transfer instructions ①, ②, and ③, i.e., the execution order is ①-A, ②-A, ③-A. Similarly, the multiple instruction segments ①-B, ②-B, and ③-B in instruction set B are executed in the order of transfer instructions ①, ②, and ③, i.e., the execution order is ①-B, ②-B, ③-B. Therefore, the execution order of the multiple instruction segments in this example could be ①-A, ②-A, ③-A, ①-B, ②-B, ③-B.

[0113] In the above description, the storage conversion module performs data transfer between the first and second buffers according to the second transfer instruction from the instruction issuance module. This application also proposes that the storage conversion module can acquire and determine, based on attributes stored by the instruction issuance module, whether to perform data transfer from the first buffer to the second buffer, or from the second buffer to the first buffer, and complete the data transfer. In one possible implementation, the storage conversion module is further configured to periodically acquire attributes of vector operation-related data recorded by the instruction issuance module, and determine changes in the activity state of the vector operation-related data within the attributes.

[0114] This approach allows the command issuing module to only record the attributes of the data related to vector operations, reducing its data processing costs. Furthermore, it provides more options for triggering data transfer, making the data processing device more flexible in its operation.

[0115] For example, the storage conversion module can periodically retrieve the starting address and data size of vector operation-related data stored in the first buffer, the activity status of the vector operation-related data, the dirty status of the vector operation-related data, and the starting address, data size, and offset of the vector operation-related data stored in the second buffer, according to a preset time interval. When the value of the activity status of the vector operation-related data changes, the storage conversion module performs data transfer between the first and second buffers.

[0116] The data transfer is divided into two types: from the first buffer to the second buffer and from the second buffer to the first buffer. These transfers occur when the data's activity status value changes from 0 to 1 and from 1 to 0, respectively.

[0117] When the activity level of data changes from 0 to 1, the storage conversion module, based on the starting address of the data whose activity level changed from 0 to 1 in the first buffer, the starting address of the data in the second buffer, and the data size, begins the transfer of the data whose activity level changed from 0 to 1. Referring to the example above, a storage address can first be allocated in the second buffer for the data whose activity level changed from 0 to 1, and then the data transfer can proceed. If the space in the second buffer is insufficient, some data can be transferred from the second buffer to the first buffer according to the category of data stored in the second buffer, and then the data whose activity level changed from 0 to 1 can be transferred to the second buffer. An exemplary implementation method can be referred to the description above, and will not be repeated here.

[0118] When the activity status value of data jumps from 1 to 0, the storage conversion module checks the dirty bit value of the data whose activity status value jumped from 1 to 0. If the dirty bit value is 1, it indicates that the data in the first buffer and the second buffer are inconsistent. The storage conversion module starts to move the data whose activity status value jumped from 1 to 0 according to the starting address of the data stored in the first buffer, the starting address of the data stored in the second buffer, the data size, and the offset. An exemplary implementation method can be referred to the description above, and will not be repeated here. If the dirty bit value is 0, it indicates that the data in the first buffer and the second buffer are consistent, and data movement from the second buffer to the first buffer is not required.

[0119] The above mainly describes the exemplary structure and operation of the data processing apparatus according to embodiments of this application. The following describes the exemplary operation of the compiler according to embodiments of this application.

[0120] In one possible implementation, embodiments of this application provide a data processing method executed by a compiler. The method includes: creating a second storage level, the second storage level corresponding to the logical storage level of the first buffer and the second buffer; and setting the second storage level to be incompatible with the logical storage level corresponding only to the first buffer and the logical storage level corresponding only to the second buffer.

[0121] For example, the compiler in this embodiment can be located on the first device. It is used to compile code written by a programmer or generated by operator code automatic generation software to obtain transmission instructions. The compiler can introduce a second storage level, Fused Unified Buffer, at the logical storage level. As described above, the second storage level corresponds to the logical storage level of the first buffer and the second buffer. The programmer can complete the program writing based on data defined in the logical storage level GlobalMemory and the second storage level Fused Unified Buffer. In this case, the maximum usable capacity of the second storage level Fused Unified Buffer is approximately 1.25MB (1MB + 256KB). Therefore, for data related to vector operations, when writing instructions, the programmer also ensures that the logically simultaneously stored data in the second storage level Fused Unified Buffer does not exceed 1.25MB. Compared with the prior art, the usable capacity for programming is greatly increased.

[0122] The compiler can configure the second storage level, Fused Unified Buffer, to be incompatible with existing logical storage level Unified Buffers that correspond only to the second buffer and logical storage level L1 Buffers that correspond to the first buffer (e.g., Fused Unified Buffer && (L1 Buffer || Unified Buffer) = 0). In this way, it can be ensured that a single operator in a vector operation unit either contains only the second storage level Fused Unified Buffer, or only logical storage level L1 Buffer or logical storage level Unified Buffer, thus guaranteeing program correctness.

[0123] In one possible implementation, the method further includes determining the category of the data related to vector operations located in the second storage level. The determined data category can be recorded in attributes by the instruction module. An exemplary method for compiler-implemented data category determination is described below.

[0124] In one possible implementation, the data categories related to vector operations in the second storage level include a first category, a second category, and a third category. The data in the second category is used more frequently than the data in the third category, and the data in the third category is used more frequently than the data in the first category.

[0125] The data of the first category is written into the data other than itself in the related data of the vector operation within the first time threshold duration. The first time threshold duration is determined according to the total duration of the first threshold instructions, or according to the total duration of the first threshold time cycles, or according to the total duration of the first threshold write or write-out operations.

[0126] The second category of data is written to itself within the second time threshold duration, which is determined based on the total duration of the second threshold instructions, or the total duration of the second threshold time cycles, or the total duration of the second threshold write or write-out operations.

[0127] The third category of data was not written into the data other than itself in the related data of the vector operation within the first time threshold duration, and was not written into itself within the second time threshold duration.

[0128] In this way, the usage frequency of data related to vector operations located in the second storage level can be determined. When the remaining storage space in the second buffer is less than the data size of the data associated with a certain type of instruction within the data related to vector operations, a criterion is provided for the data processing device to determine the data to be moved back to the first buffer from the second buffer.

[0129] For example, the compiler performs static analysis during the compilation phase on all data defined in the second storage level Fused Unified Buffer (including data related to vector operations) to determine if the data has been updated within a preset time. Based on this update, the compiler determines the data category. The methods for determining if data has been updated within the preset time and for determining the data category are described below.

[0130] Specifically, if data is written to other data (excluding itself) within the relevant data of a vector operation within the first time threshold duration, the compiler can classify it as data of the first category. The first time threshold duration can be determined based on the total duration of the first threshold instructions, the total duration of the first threshold time periods, or the total duration of the first threshold write or write operations.

[0131] For example, taking the vector addition C = A + B above as an example, vector data A, vector data B, and vector data C are different data, and vector data C can be written by vector data A and vector data B. In this case, for vector data C, if it meets the condition of being written by data other than itself (vector data C) within a first time threshold duration, such as being written by data other than itself within a total duration of less than or equal to low_limit (first threshold) instructions, or being written by data other than itself within a total duration of low_limit (first threshold) time periods, or being written by data other than itself within a total duration of low_limit (first threshold) write or write operations, then the category of vector data C can be determined to be the first category. Data in the first category can be considered as data with low usage frequency.

[0132] For example, in vector computations of the reduction type, such as C = ∑A, the vector data C is accumulated by itself, that is, written to itself. In this case, the condition "written to data other than itself" is not met for the vector data C, therefore the category of the vector data C is not the first category.

[0133] If data is written to itself within the second time threshold duration, the compiler can classify it as data of the second category. The second time threshold duration is determined based on the total duration of the second threshold instructions, or the total duration of the second threshold time cycles, or the total duration of the second threshold write or write operations.

[0134] For example, taking the vector calculation C = ∑A of the reduction type mentioned above as an example, the vector data C is written to itself. In this case, for the vector data C, if it satisfies the condition of being written to itself (vector data C) within the second time threshold duration, such as being written to itself (vector data C) within the total duration of more than or equal to high_limit (second threshold) instructions, or being written to itself (vector data C) within the total duration of high_limit (second threshold) time periods, or being written to itself (vector data C) within the total duration of high_limit (second threshold) write or write operations, then the category of the vector data C can be determined to be the second category. Data in the second category can be considered as data used more frequently.

[0135] For example, configuration information data defined in the second storage level is only written once during program initialization, and only write operations occur at other times. Therefore, it can be determined that the configuration information data belongs to the second category.

[0136] Specifically, if data is not written to other data in the related data of vector operations within the first time threshold duration, and is not written to itself within the second time threshold duration, the compiler can classify it as data of the third category. Examples of the second and first time threshold durations can be found in the description above.

[0137] For the three types of data mentioned above, the compiler can add identifiers to the relevant transmission instructions. For example, it can add the identifier ".tstream" for the first type of data, ".tconstant" for the second type, and ".tmoderate" for the third type. The instruction issuing module can determine the type of data associated with the instruction based on the identifier. This application does not restrict the identifiers used to distinguish between different types of data.

[0138] Those skilled in the art will understand that the methods for determining data categories are not limited to the examples described above. This application does not impose any limitations on the methods for determining data categories.

[0139] In one possible implementation, the method includes clustering multiple transmission instructions. An exemplary method for compiler-implemented instruction clustering is described below.

[0140] In one possible implementation, a set of transmission instructions is obtained by clustering multiple transmission instructions. The clustering is based on the write or write information between the data in the second storage level and the data in the first storage level, as well as the dependencies between the data in the relevant data of vector operations. The first transmission instruction in the set of transmission instructions obtained by clustering is given a clustering start identifier, and the last transmission instruction is given a clustering end identifier.

[0141] In this way, program segments with cluster start and cluster end identifiers can be recognized as a set of transmission instructions by the instruction issuing module of the data processing device, and processed according to the response operation to the set of transmission instructions.

[0142] For example, the compiler can cluster transfer instructions based on information such as memory access information of the data processing device (write or write information between data in the second storage level and data in the first storage level) and the dependencies between data. Specifically, based on the memory access information of the data processing device, the dependencies between data associated with at least one transfer instruction can be determined. If the dependencies can form a directed acyclic graph (see...), then... Figure 5 If we consider an example (e.g., at least one transmission instruction), then the at least one transmission instruction is clustered into a set of transmission instructions. That is, the dependencies between the data related to a set of transmission instructions can form a directed acyclic graph (DAG). By forming the dependencies of the DAG, a set of transmission instructions that can be executed simultaneously is determined, so that the data processing device can generate corresponding transport instructions based on the set of transmission instructions and transport the data involved in the transmission instructions.

[0143] Taking the transmission commands ①②③ in the above description of the command issuance module as an example, the data associated with transmission command ① are vector data C, vector data A, and vector data B. The dependency relationship determined by transmission command ① is vector data A→vector data C and vector data B→vector data C; the data associated with transmission command ② are vector data D, vector data A, and vector data C. The dependency relationship determined by transmission command ② is vector data A→vector data D and vector data C→vector data D; the data associated with transmission command ③ is vector data D. The dependency relationship determined by transmission command ③ is vector data D→vector data D. Figure 5 An example of a dependency graph determined by transmission instructions ①②③ according to an embodiment of this application is shown. Figure 5As shown, vector data D forms a cycle with itself (i.e., vector data D accesses itself), but there are no cycles between vector data A and vector data D, or between vector data B and vector data D (i.e., vector data A and B do not access vector data D). Therefore, the dependency relationship of transfer instructions ①②③ is a directed acyclic graph, and transfer instructions ①②③ can be clustered into a group of transfer instructions. For this group of transfer instructions, the compiler can add identifiers, such as adding a clustering start identifier ".group_st" at the first instruction of the group and a clustering end identifier ".group_ed" at the last instruction of the group. Optionally, for the other instructions in the group of transfer instructions besides the first and last instructions, the identifier ".group_in" can be added. An example of transfer instructions ①②③ after adding identifiers is as follows:

[0144] ①: Vec_C=Vec_A+Vec_B; .group_st;

[0145] ②: Vec_D=Vec_A*Vec_C; .group_in;

[0146] ③: Vec_D=log(Vec_D);.group_ed;

[0147] Those skilled in the art will understand that the identifier is not limited to the above examples. For instance, for transmission instructions ①②③ clustered into a group of transmission instructions, if this group of transmission instructions is the first group of transmission instructions obtained from the clustering, the same identifier ".group_A" can be added to transmission instructions ①②③. For the second group of transmission instructions obtained from the clustering, the same identifier ".group_B" can be added, so that the identifier can be used to determine whether the instructions belong to the same group. This application does not limit the way the identifier of the instructions is set.

[0148] In one possible implementation, embodiments of this application provide an instruction fetching method, which can be executed by a processor of a first device. The method includes: defining a memory scope, wherein the memory scope includes a second memory level; and setting the second memory level to be incompatible with logical memory levels corresponding only to a first buffer and logical memory levels corresponding only to the second buffer; generating code according to the memory scope, and compiling the code to obtain a transmission instruction.

[0149] For example, the storage space parameter types defined by the existing operator code auto-generation software may include a logical storage level L1 Buffer corresponding to the first buffer and a logical storage level UnifiedBuffer corresponding to the second buffer. For vector type operations, when generating code, the tunable operator template of the operator code auto-generation software only uses the logical storage level Unified Buffer space.

[0150] This application's instruction fetching method defines a new storage space parameter type, "Fused Unified Buffer," and, similar to the compiler, sets the second storage level Fused Unified Buffer in the storage space parameter type to be incompatible with the logical storage level L1 Buffer and the logical storage level Unified Buffer (Fused Unified Buffer && (L1 Buffer || Unified Buffer) = 0). In this way, for operator code automatic generation software supporting the instruction fetching method, if the tunable scheduling operator template only uses the logical storage level Unified Buffer space and not the logical storage level L1 Buffer space, the tunable scheduling operator template can automatically replace the Unified Buffer with the second storage level Fused Unified Buffer, thus obtaining a new tunable scheduling operator template. This new tunable scheduling operator template then provides the performance improvement resulting from the increased storage level capacity. Based on this, the operator code automatic generation software can generate code, which, when provided to the compiler, is used by the compiler to generate transmission instructions.

[0151] This application provides a neural network processor, including a memory module and the data processing device described above.

[0152] Neural network processors can be applied to at least one of the following terminals: mobile phones, foldable electronic devices, tablets, desktop computers, laptops, handheld computers, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, smart city devices, and server devices. This application does not impose any special limitations on the specific application of neural network processors.

[0153] The memory module stores instructions and data. In some embodiments, the memory module can be a cache memory. This memory module can store instructions or data that the data processing device has used or that are used frequently. If the memory module needs to use the instruction or data, it can be directly retrieved from the memory module. This avoids repeated accesses, reduces the waiting time of the data processing device, and thus improves system efficiency.

[0154] A memory module can be used to store executable program code, including instructions. The memory module may include a stored program area and a stored data area. The stored program area may store the operating system, at least one application program required for a function (such as transfer instructions), etc. The stored data area may store data created during the use of the data processing device (such as the results of operations by vector processing units), etc. Furthermore, the memory module may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. The data processing device executes various functional methods of the data processing device by running the instructions stored in the memory module.

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0156] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware (such as circuits or ASICs (Application Specific Integrated Circuits)) that performs the corresponding function or action, or using a combination of hardware and software, such as firmware.

[0157] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0158] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data processing apparatus, characterized in that, This is applied to a neural network processor, which further includes a memory module and a vector operation unit. The data processing device includes a first buffer, a second buffer, a storage conversion module, and an instruction issuing module. The memory module stores data related to vector operations. The instruction issuing module is used to generate a first transport instruction and a second transport instruction; The storage conversion module is used to move the relevant data of the vector operation from the memory module to the first buffer according to the first moving instruction, and to move the relevant data of the vector operation from the first buffer to the second buffer according to the second moving instruction, wherein the capacity of the first buffer is larger than that of the second buffer, and the vector operation unit obtains the relevant data of the vector operation from the second buffer; The generation of the first and second transfer instructions includes: recording attributes of the data related to the vector operation; when a transfer instruction is received from the memory module, generating the first transfer instruction based on the attributes; when multiple transfer instructions are received from the memory module, for each set of transfer instructions between each transfer instruction containing a cluster start identifier and the first subsequent transfer instruction containing a cluster end identifier, determining the storage address in the second buffer of the data associated with the set of transfer instructions in the data related to the vector operation in the set of transfer instructions, wherein the set of transfer instructions is executed simultaneously; and generating the second transfer instruction based on the attributes and the storage address.

2. The apparatus according to claim 1, characterized in that, The transmission instruction indicates that the data related to the vector operation be changed from the first storage level to the second storage level, where the first storage level corresponds to the logical storage level of the memory module, and the second storage level corresponds to the logical storage level of the first buffer and the second buffer.

3. The apparatus according to claim 2, characterized in that, The attributes include at least one of the following: whether the data related to the vector operation is defined in the transmission instruction; the starting address and size of the data related to the vector operation stored in the first buffer; the activity status of the data related to the vector operation; the dirty status of the data related to the vector operation; the category of the data related to the vector operation; the usage frequency index of the data related to the vector operation; and the starting address, size, and offset of the data related to the vector operation stored in the second buffer. Whether it is defined in the transmission instruction indicates whether it is a declared variable in the transmission instruction; the active state indicates the storage state of the first buffer or the second buffer; the dirty bit indicates whether the data stored in the first buffer and the second buffer are consistent; and the category indicates whether the data has been updated within a preset time period.

4. The apparatus according to claim 3, characterized in that, Generating the first transport instruction based on the attribute includes: The first transport instruction is generated based on the starting address and data size of the relevant data stored in the first buffer according to the vector operation.

5. The apparatus according to claim 3 or 4, characterized in that, The command issuing module is also used for: Based on the current storage location of the data related to the vector operation, update at least one of the following attributes: the starting address and data size of the data related to the vector operation stored in the first buffer; the activity status of the data related to the vector operation; and the starting address, data size, and offset of the data related to the vector operation stored in the second buffer.

6. The apparatus according to any one of claims 2-4, characterized in that, The instruction issuing module is further configured to, when the remaining storage space in the second buffer is less than the data size of the data associated with the set of transmission instructions in the relevant data of the vector operation, generate a third transfer instruction based on the attributes of the data stored in the second buffer. The third transfer instruction instructs to transfer a portion of the data stored in the second buffer from the second buffer to the first buffer. The sum of the data size of the portion of data and the remaining storage space in the second buffer is greater than or equal to the data size of the data associated with the set of transmission instructions in the relevant data of the vector operation. The storage conversion module is also used to move the portion of data from the second buffer to the first buffer according to the third transfer instruction.

7. The apparatus according to claim 6, characterized in that, The data stored in the second buffer has attributes including categories, and different categories of data are used with different frequencies. When generating the third transport instruction, the usage frequency corresponding to the category of the partial data is less than or equal to the usage frequency corresponding to the category of the remaining data in the second buffer excluding the partial data.

8. The apparatus according to any one of claims 2-4, characterized in that, The command issuing module is also used for: Each transmission instruction in the set of transmission instructions is divided into multiple instruction segments corresponding to each transmission instruction. Instructions with the same division order among the multiple instruction segments corresponding to each transmission instruction are combined to form multiple instruction sets, and the division order of different instruction sets is different. The plurality of instruction sets are executed, wherein each instruction set in the plurality of instruction sets is executed in the order of instruction segmentation, and each instruction in each instruction set is executed in the order of its corresponding transmission instruction in the set of transmission instructions.

9. The apparatus according to claim 3 or 4, characterized in that, The storage conversion module is also used to periodically acquire the attributes of the vector operation-related data recorded by the instruction issuance module, and determine the change in the activity state of the vector operation-related data in the attributes.

10. The apparatus according to any one of claims 2-4, characterized in that, The set of transmission instructions is obtained by clustering multiple transmission instructions. The clustering is based on the write or write information between the data located in the second storage level and the data located in the first storage level, as well as the dependency relationship between the data in the relevant data of the vector operation. The first transmission instruction in the set of transmission instructions obtained by clustering is given a clustering start identifier, and the last transmission instruction is given a clustering end identifier.

11. The apparatus according to claim 10, characterized in that, The dependencies between the data related to the set of transmission instructions can form a directed acyclic graph.

12. The apparatus according to claim 3 or 4, characterized in that, The data categories related to vector operations in the second storage level include Category 1, Category 2, and Category 3. Category 2 data is used more frequently than Category 3 data, and Category 3 data is used more frequently than Category 1 data. The data of the first category is written by data other than itself in the relevant data of the vector operation within the first time threshold duration. The first time threshold duration is determined according to the total duration of the first threshold instructions, or according to the total duration of the first threshold time cycles, or according to the total duration of the first threshold write or write-out operations. The second category of data is written to itself within the second time threshold duration, which is determined based on the total duration of the second threshold instructions, or the total duration of the second threshold time cycles, or the total duration of the second threshold write or write-out operations. The third category of data was not written by data other than itself in the relevant data of the vector operation within the first time threshold duration, and was not written by itself within the second time threshold duration.

13. A neural network processor, characterized in that, It includes a memory module and a data processing apparatus according to any one of claims 1-12.

Images