Data handling device and processor supporting multithreading

Abstract

The present application provides a data handling device supporting multi-threading, which includes a control module, an instruction cache, a context cache, and a data path running in a pipeline manner consisting of a read-in DMA module, a data cache module, and an output DMA module. After processing an instruction, each module in the data path passes the relevant instruction information and data to the next module in the pipeline, and then starts processing the next instruction; the context information related to the instruction is stored in the context cache and shared by each module. The data handling device improves the efficiency of data movement in a multi-threaded scenario while minimizing area overhead.

Description

Technical Field

[0001] The present application relates to a parallel computing processor, and more particularly to a device and processor suitable for data transfer in multi-core parallel computing. Background Art

[0002] The statements in this section are merely intended to provide background information related to the technical solution of the present application to aid understanding, and they do not necessarily constitute prior art for the technical solution of the present application.

[0003] Artificial intelligence (AI) technology has developed rapidly in recent years and has penetrated into various fields such as visual perception, speech recognition, assisted driving, smart home, traffic dispatch, etc. Many AI algorithms involve learning and computing based on neural networks, such as convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), etc. These AI algorithms require strong parallel computing capabilities to process massive data, so processors that support multi-core parallel computing such as GPUs, GPGPUs, and AI accelerators are usually used to perform multi-layer neural network operations. These processors usually use a single instruction multiple threads (SIMT) architecture, and tasks of multiple threads can run simultaneously on different computing cores of the processor. When a group of threads is waiting due to memory access or other reasons, it is suspended and another group of threads is executed instead. The control unit of the processor can quickly switch between multiple groups of threads to maximize the throughput of the processor. However, the access speed of data has always been an important factor affecting the computing efficiency of the processor. How to efficiently carry out large amounts of data between on-chip cache and off-chip memory is an urgent problem to be solved in parallel computing processors.

[0004] It should be noted that the above content is only used to help understand the technical solution of the present application, and is not used as a basis for evaluating the prior art of the present application. Summary of the invention

[0005] The inventors found in practice that although the access speed of data can be improved by setting up an on-chip cache inside the chip to pre-fetch part of the data and cache the calculation results, the capacity of the on-chip cache is usually limited due to limitations such as cost and volume. Especially in applications involving neural networks, such limited on-chip caches often cannot cache all data related to multiple layers of the neural network. In addition, the inventors also tried to use direct memory access (DMA) to move data between the on-chip cache and the off-chip memory, but found that each time DMA is enabled, each computing core needs to start and configure DMA transmission separately, which greatly reduces the execution efficiency of a large number of multi-threaded tasks on multiple cores. Although it is possible to try to improve the parallelism between instructions through the DMA of the pipeline architecture, in SIMT processors used to execute AI algorithms such as neural networks, in addition to instructions, additional contexts are often required to describe the data to be processed (such as the data arrangement of the input and output of each layer of the neural network, data type, feature map size, etc.), if these contexts are propagated along the pipeline together with the instructions, it will inevitably cause a large area consumption.

[0006] Therefore, an embodiment of the present application provides a data transfer device that supports multi-threading, which improves the data transfer efficiency in a multi-threading scenario while minimizing area overhead, and is particularly suitable for data transfer in a SIMT processor.

[0007] The above purpose is achieved through the following technical solutions:

[0008] According to a first aspect of an embodiment of the present application, a data handling device supporting multithreading is provided, comprising a control module, an instruction cache, a context cache, and a data path running in a pipeline manner. The data path at least comprises a read-in DMA module, a data cache module, and an output DMA module. The instruction cache and the context cache can be shared by each module in the data path, and wherein:

[0009] The control module is used to store the received instructions to be processed in the instruction cache, and to store the context information related to the instructions to be processed in the context cache. The read-in DMA module is used to obtain the instructions to be processed from the instruction cache and read the data to be transported according to the context information related to the instruction obtained from the context cache, and pass the read data together with the corresponding instruction information to the next module in the data path. The data cache module is used to select the corresponding context information from the context cache according to the corresponding instruction information received, cache the received data and the corresponding instruction information according to the obtained context information, and pass the cached data and its corresponding instruction information to the next module in the data path when the cached data meets the output requirements. The output DMA module is used to select the corresponding context information from the context cache according to the corresponding instruction information received, generate the output address according to the obtained context information, and output the output address together with the received data.

[0010] In this embodiment, each module in the data path can be configured to pass the data involved in the instruction and corresponding instruction information to the next module in the data path after processing the current instruction, and then start processing the next instruction, wherein the corresponding instruction information is used to identify the current instruction and its related context information.

[0011] In some embodiments, the read DMA module can also be configured to: while passing the read data and the corresponding instruction information to the next module in the data path, feedback instruction completion information to the control module and obtain the next instruction to be processed from the instruction cache. The output DMA module can also be configured to: while outputting the output address and the received data together, feedback instruction completion information to the control module. The control module can also be configured to clear the completed instructions from the instruction cache in response to the instruction completion information from the read DMA module, and clear the context information related to the completed instructions from the context cache in response to the instruction completion information from the output DMA module.

[0012] In some embodiments, the data path may further include an arithmetic logic unit module before the data cache module, which selects corresponding context information from the context cache according to the corresponding instruction information received from the read-in DMA module, performs a preset operation on the data received from the read-in DMA module according to the acquired context information, and passes the processed data and the corresponding instruction information to the data cache module. In some embodiments, the preset operation may include one of the following: calculation of activation functions in neural networks, data type conversion.

[0013] In some embodiments, the data cache module may be configured to perform a corresponding transformation operation on the received data to obtain output data according to the description of the output data in the acquired context information, and cache the transformed data and the corresponding instruction information. The transformation operation may include one or more of the following combinations: rotation, inversion, transposition, merging, and splitting.

[0014] In some embodiments, the control module may be further configured to receive status information related to the instruction, and set or adjust the context information of the corresponding instruction in the context cache according to the received status information.

[0015] In some embodiments, the data handling device also includes an instruction interface, a configuration interface and a status interface, wherein the instruction interface is used to receive instructions; the configuration interface is used to receive context information related to the instructions; and the status interface is used to receive status information related to the instructions.

[0016] In some embodiments, the corresponding instruction information transmitted along the pipeline with the data in the data path includes at least an instruction identifier. Each module in the data path can identify the corresponding instruction from the instruction cache according to the instruction identifier, and identify context information related to the instruction from the context cache.

[0017] In some embodiments, the corresponding instruction information transmitted along the pipeline together with the data in the data path includes at least an instruction identifier and a context identifier. Each module in the data path can identify the corresponding instruction from the instruction cache according to the instruction identifier, and identify the context information related to the instruction from the context cache according to the context identifier. The control module can also be configured to establish an association relationship between the instruction to be processed stored in the instruction cache and the relevant context information of the instruction stored in the context cache by setting a mapping between the instruction identifier and the context identifier. In this way, each module in the data path can identify the context information related to the currently processed instruction from the context cache according to the established association relationship.

[0018] In some embodiments, the context information may include a combination of one or more of the following: input data arrangement information, output data arrangement information, input data data type, output data data type, arithmetic logic unit module control information, feature map size information, data expansion information, region of interest information, data filling information, data pooling information, and handshake granularity information.

[0019] According to a second aspect of the embodiment of the present application, a processor is provided, comprising the multi-threaded data handling device according to the first aspect of the embodiment of the present application, which is used to transfer data between the on-chip cache and the off-chip memory of the processor. Such a processor is particularly suitable for neural network computing.

[0020] The technical solution of the embodiment of the present application may have the following beneficial effects:

[0021] The multi-threaded data handling device improves the data handling efficiency in the scenario of supporting multi-threaded parallel computing on multiple cores while minimizing the area overhead. In addition, the multi-threaded data handling device also has certain data processing capabilities in addition to data handling, which improves the adaptation efficiency of input and output data between layers of the neural network, and is particularly suitable for data handling of SIMT architecture processors (such as GPU, GPGPU and AI accelerators) for neural networks.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0024] Figure 1 A schematic diagram of the structural modules of a data handling device supporting multi-threading according to an embodiment of the present application.

[0025] Figure 2 This is a schematic diagram of the structural modules of a data handling device supporting multi-threading according to yet another embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of contradictions or conflicts, the various embodiments provided in the present application can be combined with each other, and some or all of the features in the various embodiments can be used in combination with each other.

[0027] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0028] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0030] In processors that support multi-threaded operation, such as SIMT processors, multiple data of the same instruction are allowed to be addressed separately, multiple threads are supported to execute the same instruction on different data sets, and different thread tasks can be executed on different computing cores. Therefore, for the same instruction, there may be different contexts to describe the data processed by the instruction and the information required to execute the instruction. In particular, when such processors are used in neural network operations, the volume of context information used to describe the data to be processed by the instruction is relatively large; for example, the data arrangement, data type, number of channels, feature map size, etc. of the input and output of each layer of the neural network are required. Therefore, in the scenario of multi-threaded parallel execution, these context information occupy a lot of bandwidth overhead during data transfer.

[0031] In an embodiment of the present application, a data handling device supporting multi-threading is provided, which separates instruction information from context information related to the instruction. This can not only simplify the instructions, but also facilitate flexible control of the context information of the instructions. It is particularly suitable for SIMT architecture processors that support multiple threads to execute the same instructions on different data sets.

[0032] Figure 1 A functional module diagram of a data handling device supporting multi-threading according to one embodiment of the present application is provided.

[0033] The data handling module comprises: a control module, an instruction cache, a context cache and a data path running in a pipeline manner consisting of a read-in DMA module, a data cache module and an output DMA module.

[0034] The control module is used to: store the received one or more instructions to be processed and their related context information in the instruction cache and the context cache respectively.

[0035] The context information related to the instruction generally refers to all relevant information required to complete the instruction, and may include, for example, information related to instruction execution, information related to data processed by the instruction, etc. Different instructions have different context information, and this document does not impose any limitation on this.

[0036] The instruction cache can be shared by all modules in the data path.

[0037] In the embodiment of the present application, the context cache is shared by various modules in the data path.

[0038] When processing an instruction, each module in the data path can extract context information related to the instruction from the context cache; after completing the processing of an instruction, the data involved in the instruction and the corresponding instruction information can be transmitted to the next module in the data path, and the next instruction can be obtained for processing. The corresponding instruction information transmitted in the data path together with the data here is: information required for each level of modules in the data path to identify the corresponding instruction to be processed and its context (for example, instruction identifier, context identifier, etc.), and does not include specific context information related to the instruction. In this data handling device, the data path running in a pipeline manner transmits data and corresponding instruction information along the pipeline but does not transmit context information, thereby improving the data handling efficiency while minimizing the area overhead.

[0039] More specifically, refer to Figure 1When the read-in DMA module receives an instruction from the control module or extracts an instruction to be processed from the instruction cache, the context information corresponding to the instruction is found from the context cache module. The read-in DMA module can generate a data request to an external storage module based on the context information related to the instruction obtained. The data request at least includes the address and size of the data to be read. After receiving the data request, the external storage module returns the data to the read-in DMA module. The read-in DMA module can output the received data together with the relevant information of the currently processed instruction (for example, the instruction identifier) ​​to the next module in the data path, such as the data cache module. After the read-in DMA module completes the transmission, the read-in DMA module can immediately feedback the instruction completion information to the control module, and obtain the next instruction from the instruction cache, and at the same time obtain the context information of the instruction in the context cache module, so that the operation of the next instruction can be started.

[0040] Optionally, after receiving the instruction completion information from the read-in DMA module, the control module can clear the completed instruction from the instruction cache to prepare for receiving the next instruction. However, the context information related to the completed instruction stored in the context cache module still needs to be retained for use by subsequent modules in the pipeline. In some embodiments, since the volume of the instruction itself is very small (the volume of the instruction information used to indicate the transported data is small), compared to the method of transmitting the instruction itself and the specific context information corresponding to the instruction along the pipeline, in the case where the pipelined data path does not directly transmit the specific context information, even if the instruction itself is transmitted along the pipeline, it will not cause a large area overhead. Each module in the data path can choose to transmit the current instruction itself and / or the instruction identifier of the current instruction along the pipeline, but the pipelined data path still does not directly transmit the context information of the instruction, but each module in the data path obtains the context information required by each module from the shared context cache according to the corresponding instruction information received and the current instruction and / or instruction identifier. Among them, if the pipeline does not pass the specific context corresponding to the instruction, nor does it directly pass the instruction itself, but instead transfers data by passing an identifier and obtaining the cache content based on the identifier, the instructions can be streamlined and the bandwidth overhead occupied during data transfer can be further reduced.

[0041] The data cache module can be used to receive data and corresponding instruction information output from the upper-level module in the data path (for example, it can be a read-in DMA module). The data cache module can select corresponding context information from the context cache according to the corresponding instruction information received, and cache the received data and corresponding instruction information according to the acquired context information. When the data meets the output requirements, the data cache module can pass the cached data and its corresponding instruction information to the next-level module in the data path (for example, the output DMA module). For example, the data cache module can start outputting after detecting that there is enough data in the cache, or it can start outputting when the amount of cached data can meet the basic data requirements of the output channel. For another example, the data cache module can merge the data according to the relevant information in the context information to increase the bandwidth utilization of the output data.

[0042] In some embodiments, the data cache module can also perform corresponding transformation operations on the received data to obtain corresponding output data according to the description information about the output data in the acquired context information, and cache the transformed data and the corresponding instruction information. The transformation operation here may include but is not limited to one or more combinations of the following: rotation, reversal, transposition, merging, splitting, etc. For example, the data cache module can perform transformation operations such as rotation, reversal and transposition of different latitudes on the received data according to the description information such as the input data arrangement information and the output data arrangement information contained in the context information, so that the input data can better adapt to the format of the output data. This is particularly suitable for the processing of multi-layer neural networks, because the input data dimensions, formats or types required by different network layers may be different, and the data handling module will have higher adaptability when performing transformation operations in the process of caching data. When the processor finishes processing a network layer, the processing result will be written into the cache and saved to the memory through the data handling module, and when starting the processing of the next network layer, the data can be directly read without performing data format conversion. This is because the data handling module has completed the corresponding data conversion and adaptation work in the process of saving data. This obviously reduces the computing load of the processor and is conducive to improving the throughput of the processor for large data.

[0043] Continue to refer Figure 1The output DMA module receives data and corresponding instruction information output from the upper-level module (e.g., data cache module) in the data path, and selects context information associated with the instruction from the context cache according to the received corresponding instruction information. The output DMA module can generate an output address according to the acquired context information, and output the output address together with the received data, for example, writing the received data to the corresponding address in the target memory. At the same time, the output DMA module can feedback instruction completion information to the control module. The control module clears the context information related to the instruction from the context cache in response to the instruction completion information from the output DMA module. The control module can also send a notification to an external related control unit to indicate that the data handling device is ready to receive the next instruction.

[0044] In some embodiments, in the data path of the data handling device, an arithmetic logic unit module (which may be referred to as an ALU module) located before the data cache module may also be included. The ALU module selects context information related to the instruction from the context cache according to the corresponding instruction information received from the read-in DMA module, and performs a preset operation on the data received from the read-in DMA module according to the context information obtained. Such preset operations include but are not limited to: calculation of activation functions in neural networks, data type conversion, etc. The ALU module passes the processed data and the corresponding instruction information together to the next module in the data path (for example, it can be passed to the data cache module). It should be understood that the ALU module here is an optional module, which is particularly suitable for neural network processing. For example, in a multi-layer neural network, the output feature data of the previous layer is provided to the next layer as the input feature data of the next layer for further processing. Each layer often operates the input neuron vector and the corresponding weight vector, and then the obtained operation result is passed through the activation function to generate the output feature. Such an activation function will be repeated in each layer. Therefore, in the embodiment of the present application, an ALU module is provided in the data handling device to process the activation function, so that the execution of these repeated activation functions is independent of each computing core. In this way, after processing the calculation of one layer of the neural network, the calculation results can be directly saved in the corresponding on-chip cache. The data handling device can complete the execution of the activation function during the handling of these intermediate results and save the data processed by the activation function; and when starting the next layer of the neural network, the computing core can directly obtain the saved data for calculation. Through such an ALU module, some repeated fixed operations can be performed, reducing the computing load of the computing core and improving the data processing throughput of the processor.

[0045] In some embodiments, the control module may also receive state information related to the instruction, and set or adjust the context information of the corresponding instruction in the context cache according to the received state information. This is in view of the fact that in processors supporting multi-threaded operation such as SIMT processors, there are often multiple threads executing the same instruction on different data sets, or multiple threads on different computing cores executing different instructions on the same or similar data sets. Thus, in thread switching, many parts of the context information related to multiple instructions remain unchanged, and only a small part of the information changes. By updating some contents in the context cache through the state information, the configuration flexibility and processing efficiency can be improved. In some other embodiments, in order to configure the context information more flexibly, the control module may actively send a state information request to an external control unit, and update and adjust the corresponding context information in the context cache according to the received state information. The control module may also receive state information actively sent by an external control unit to adjust and set certain data items in the corresponding context information.

[0046] In some other embodiments, the data handling device further includes a command interface, a configuration interface, and a status interface. The control module receives commands from the outside through the command interface, receives context information related to the commands through the configuration interface, receives status information related to the commands through the status interface, and issues a status information request to an external control unit. In this way, the data handling device can receive status information or issue a status request to obtain status information through a dedicated status interface, thereby using updated status information to fill in or set data items that frequently change in the context information, thereby achieving a more flexible configuration method.

[0047] In the above-mentioned embodiment, the data path of the data handling device operates in a pipeline mode. Each module in the data path transmits data and corresponding instruction information along the pipeline, so each module can immediately obtain the next instruction for processing after processing an instruction, thereby improving the parallel degree of the instruction. In one example, the corresponding instruction information transmitted along the pipeline with the data in the data path includes at least an instruction identifier. Through the instruction identifier, each module in the data path can identify the corresponding instruction from the instruction cache, and can also identify the context information related to the instruction from the context cache. In another example, the relevant information of the instruction transmitted along the pipeline with the data in the data path includes at least an instruction identifier and a context identifier. The instruction identifier is used to identify the corresponding instruction from the instruction cache. The context identifier is used to identify the context information related to the instruction from the context cache. In this example, the control module can establish a flexible correspondence between the instruction to be processed stored in the instruction cache and the relevant context information stored in the context cache by setting the mapping between the instruction identifier and the context identifier. For example, one instruction can correspond to different context information, and the same context information can correspond to different instructions. Each module in the data path determines the context information corresponding to the currently processed instruction through the instruction identifier and the context identifier. This can support more diverse processing scenarios.

[0048] In an embodiment of the present application, a data handling device supporting multithreading is provided. The data handling device distinguishes different types of instructions or threads through instruction identifiers (instruction IDs for short), so that they can be run in the data handling device at the same time to improve the efficiency of data handling. The number of instructions supported to run simultaneously depends on the number of instructions that may exist simultaneously in the data path. And in the data handling device, part of the instruction information such as the instruction identifier is passed along the pipeline in the data path together with the data, without the need to pass all the context information related to the instruction in the pipeline, thereby reducing the chip area overhead while improving the data handling efficiency.

[0049] Figure 2 A functional module diagram of a data handling device according to another embodiment of the present application is given in FIG. In this embodiment, the context information related to the instruction is divided into two parts: the part that does not change frequently is set as descriptor information, and the part that changes frequently is set as status information. The data handling device includes a control module composed of an instruction processing module, a descriptor processing module, a status information processing module, a merge decoding module and a context cache module, and a data path that runs in a pipeline manner composed of a read-in DMA module, an ALU module, a data cache module and an output DMA module. The following is combined with Figure 2A more detailed introduction is given to each module of the data handling device.

[0050] The data handling device receives one or more instructions distributed from an external control unit (such as a control unit of a processor, each core in a processor, etc.). The data handling device can cache the received instructions through an instruction processing module. The received instructions should be instructions that require the participation of the data handling device, and may include but are not limited to storage type instructions. The received instructions should include an instruction identifier (which may be referred to as an instruction id) to distinguish different instructions. The received instructions may also include a context identifier (which may be referred to as a context id) and description information of the data to be processed. The description information of the data to be processed contained in the instruction includes at least the starting data block address information, the target data block address information and the size of the data block. In some embodiments involving neural network operations, each layer of the neural network has different input and output feature data, and different operations are performed using different parameters. Since the number of nodes in each layer may be different, the number of channels and dimensions of the input and output data will also be different. Therefore, the description information of the data to be processed contained in the instruction may include the starting data block address information, the target data block address information, the physical address mapping information, the number of input and output data channels, the dimension size information of each input channel data block, the dimension coordinate information of each input channel data block, the dimension boundary information of each input channel data block, the storage unit indication information of each input channel, the network layer identification of the data to be processed, and the instruction end indication information, etc. The information contained in the above instructions can be obtained by decoding the received instructions in the combined decoding module, and can be partially or completely stored in the context information identified by the context id of the instruction.

[0051] In an embodiment of the present application, the instruction information and the context information describing the data are independent of each other and are input into the data handling device by the external control unit through different interfaces. As mentioned above, the context information describing the data can be divided into descriptor information and state information, but can also be all included in the descriptor information. The context information can be provided to the data handling device in the form of descriptor information and / or state information and cached in the descriptor information processing module and / or the state information processing module. The data handling module can cache multiple groups of descriptor information through the descriptor information processing module and cache multiple groups of state information through the state information processing module. The instruction received by the data handling device may also include a descriptor identifier (which may be referred to as a descriptor id) for selecting corresponding descriptor information from the descriptor information cached in the descriptor information processing module and giving it to the merged decoding module for decoding. Part or all of the context information may be included in the descriptor information corresponding to the instruction, and each descriptor information corresponds to a descriptor id. The corresponding descriptor information can be found according to the descriptor id contained in the instruction.

[0052] In the embodiment where the context information is divided into descriptor information and state information, the data handling device further comprises a state information processing module. The state information processing module initiates a state information request to the external control unit according to the context ID provided by the instruction, and caches the returned state information according to the context ID. The data handling device can cache multiple sets of state information and context IDs through the state information processing module. The state information may include part or all of the context information, and the corresponding state information can be found according to the context ID included in the instruction.

[0053] The above instructions, descriptor information and status information will all pass through the merge decoding module, and the decoded data will be merged into the context information required to execute the instruction and cached in the context cache module, identified by the context id of the instruction. The context information required for the instruction can be located from the context cache module through the instruction id and the context id. The context information should include but is not limited to input data layout information, output data layout information, input data data type, output data data type, ALU module control information, feature map size information, data dilation information, area of ​​interest information, data padding information, data pooling information, handshake granularity information, etc.

[0054] Continue to refer Figure 2, the data handling device further comprises a read-in DMA module. When the read-in DMA module receives an input instruction and determines that the required context information is in the context cache module, it will generate a data request to an external storage module. The read-in DMA module can generate a data request for reading data according to the context information related to the instruction. The data request at least includes the address and size of the data to be read. After receiving the data request, the external storage module returns the data to the read-in DMA module. After the read-in DMA module appropriately processes the data according to the context information, it outputs the data together with the instruction id and part of the instruction information (for example, the context id corresponding to the instruction, etc.) when the data request is issued to the next module in the pipeline, such as the ALU module. In this embodiment, the data read by the read-in DMA module, the instruction id corresponding to the read data and the context id are transmitted to the subsequent module together. After the transmission is completed, the instruction completion information can be immediately fed back to the instruction processing module, and the next instruction id and corresponding instruction information can be obtained. At the same time, the context information of the switched instruction can be read in the context cache module according to the instruction id, so that the operation of the next instruction can be started. After receiving the instruction completion information read into DMA, the instruction processing module can clear the completed instruction from the cache to prepare for receiving the next instruction. However, the context information related to the completed instruction saved in the context cache module still needs to be retained for use by subsequent modules in the pipeline.

[0055] The ALU module receives the data output by the DMA module, the instruction id corresponding to the data, and the context id. The ALU module can select the corresponding context information in the context cache module according to the instruction id and the context id, and perform corresponding calculations on the data according to the preset calculation information in the context information. Such calculations may include but are not limited to activation functions (such as ReLU, Tanh, SigMod) and data type conversion operations (such as conversion from FP32 to FP16, conversion from Float to Int, and conversion from FP32 to BF16) in the neural network. The ALU module will continue to pass the calculated result data, the corresponding instruction id and the context id to the next module in a pipeline manner, and the newly received data can be processed after the transfer is completed. It should be understood that the ALU module here is an optional module, especially suitable for neural network processing. For example, in a multi-layer neural network, the output feature data of the previous layer is provided to the next layer as the input feature data of the next layer for further processing. Each layer often operates the input neuron vector and the corresponding weight vector, and the obtained operation result is passed through the activation function to generate the output feature. Such an activation function will be repeated in each layer. Therefore, in the embodiment of the present application, by providing an ALU module in the data handling device to process the activation function, the execution of these repeated activation functions is independent of each computing core, so that each time the processor completes the calculation of a layer of the neural network, the calculation result can be directly saved in the corresponding on-chip cache, and the data handling device can execute the activation function during the transportation of these intermediate results, and save the output result after the activation function processing; and when starting the next layer of the neural network, the computing core directly reads the saved data for calculation. Through such an ALU module, the computing load of the computing core is reduced and the data processing throughput of the processor is improved.

[0056] The data cache module receives the data output by the ALU module, the instruction ID corresponding to the data, and the context ID. The data cache module can select the corresponding context information in the context cache module according to the instruction ID and the context ID, and write the received data into the internal cache in a certain way according to the obtained context information. For example, the data cache module can write the data into the internal cache in a specific way according to the input data arrangement information, the output data type, the cache free information, etc. For another example, the data cache module can perform operations such as rotation, reversal, and transposition of different latitudes on the received data according to different input data arrangement information and output data arrangement information, so that the input data can better adapt to the format of the output data. This is particularly suitable for the processing of multi-layer neural networks. The input data dimensions, formats, or types required for different network layers may be different. When the processor finishes processing a network layer, the processing results will be written into the cache and saved to the memory through the data handling module. When starting the processing of the next network layer, the data can be directly read without performing data format conversion; this is because the data handling module has completed the corresponding data conversion and adaptation work in the process of saving data. This obviously reduces the computational load of the processor and is conducive to improving the throughput of the processor for large data.

[0057] When the data cache module writes data into the internal cache, the instruction id and context id corresponding to the data will also be cached accordingly. The data cache module can obtain context information related to the instruction based on the cached instruction id and context id, and extract settings about the output data (such as output requirements, output data format, output bandwidth, etc.) from the context information. When the data meets the set output requirements, it immediately starts to read data from the internal cache and passes the read data, the instruction id and context id corresponding to the data to the next module in the pipeline (such as the output DMA module). For example, the data cache module can start outputting after detecting that there is enough data in the output waiting cache. For another example, the data cache module can merge the data according to the output bandwidth information in the context information to increase the bandwidth utilization of the output data.

[0058] The output DMA module receives the data output by the data cache module, the instruction id corresponding to the data, and the context id. The output DMA module can select the corresponding context information in the context cache module according to the instruction id and the context id, and generate the address of the data to be output according to the context information, and send the data to be output and the address together to the external storage module. The output DMA module clears the context information corresponding to the instruction id and the context id in the context cache module when the last data of an instruction is written out, and allows the next instruction corresponding to the instruction id and the corresponding descriptor information and status information to be written into the context cache module. While executing the operation of clearing the context cache corresponding to the instruction id, the data handling device can notify the external control module through the handshake information containing the instruction id that it can start scheduling the next instruction corresponding to the instruction id.

[0059] In some embodiments, the output DMA module may also use a multi-channel mechanism to output data in parallel. The output DMA can generate the corresponding addresses and data of the output data for different output data channels according to the context information and send them to the external storage module together. The address of the output data corresponding to each data channel can be calculated and generated according to the coordinate information of each dimension in the context information, the size information of each dimension, the size information of the feature map, the data pooling information, the output data arrangement information and the output data type information. In some other embodiments, the output DMA module also needs to consider the problem that the width of the output data spans multiple storage space boundaries, and the output data can be divided into multiple groups of outputs according to the memory space spanned by the address. In some other embodiments, the output DMA module can also generate handshake information corresponding to the layer id information when the output data volume is greater than the handshake granularity information based on the handshake granularity information in the context information and the layer id information in the instruction, so as to notify the external control module that it can start using the data corresponding to the output layer id information.

[0060] In some other embodiments of the present application, a processor for a neural network is also provided, which includes the above-mentioned Figure 1 and / or Figure 2 In this processor, tasks of multiple threads are simultaneously executed on different computing cores of the processor. Different computing cores perform different calculations according to instructions. The data processed by each computing core and the calculation results are temporarily stored in the internal on-chip cache. The multi-threaded data handling device introduced above is used to execute data transfer between the on-chip cache of the processor and the off-chip memory.

[0061] References in this specification to "various embodiments," "some embodiments," "one embodiment," or "an embodiment," etc. refer to a particular feature, structure, or property described in conjunction with the embodiment being included in at least one embodiment. Thus, the appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., in various places throughout the specification do not necessarily refer to the same embodiment. Furthermore, particular features, structures, or properties may be combined in any suitable manner in one or more embodiments. Thus, particular features, structures, or properties shown or described in conjunction with one embodiment may be combined in whole or in part with features, structures, or properties of one or more other embodiments without restriction, as long as the combination is not illogical or inoperable.

[0062] The expressions of "including", "having" and similar meanings in this specification are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices. "One" or "an" does not exclude multiple situations. In addition, the elements in the drawings of this application are for illustrative purposes only and are not drawn to scale.

[0063] Although the present application has been described through the above-mentioned embodiments, the present application is not limited to the embodiments described herein, and also includes various changes and modifications made without departing from the scope of the present application.

Claims

1. A data handling device supporting multithreading, comprising a control module, an instruction cache, a context cache and a data path running in a pipeline manner, wherein the data path comprises an input DMA module, a data cache module and an output DMA module, wherein the context cache is shared by each module in the data path, and wherein: The control module is used to store the received instructions to be processed in the instruction cache, and store the context information related to the instructions to be processed in the context cache; A read-in DMA module is used to obtain the instruction to be processed from the instruction cache and read the data to be moved according to the context information related to the instruction obtained from the context cache, and pass the read data together with the corresponding instruction information to the next module in the data path, wherein the corresponding instruction information is used to identify the current instruction and its related context information; A data cache module, used to select corresponding context information from a context cache according to the received corresponding instruction information, cache the received data and the corresponding instruction information according to the obtained context information, and pass the cached data and its corresponding instruction information to the next module in the data path when the cached data meets the output requirements; The output DMA module is used to select corresponding context information from the context cache according to the corresponding instruction information received, generate an output address according to the obtained context information, and output the output address together with the received data.

2. The device according to claim 1, wherein: The read-in DMA module is further configured to: while passing the read data together with the corresponding instruction information to the next module in the data path, feed back instruction completion information to the control module and obtain the next instruction to be processed from the instruction cache; The output DMA module is further configured to: feed back instruction completion information to the control module while outputting the output address together with the received data; The control module is further configured to: clear completed instructions from the instruction cache in response to instruction completion information from the read DMA module, and clear context information associated with the completed instructions from the context cache in response to instruction completion information from the output DMA module.

3. The apparatus according to claim 1, further comprising an arithmetic logic unit module in the data path; The arithmetic logic unit module is used to: select corresponding context information from the context cache according to the corresponding instruction information received from the read-in DMA module, and perform preset operations on the data received from the read-in DMA module according to the acquired context information, and pass the processed data and the corresponding instruction information to the data cache module.

4. The device according to claim 3, wherein the preset operation comprises at least one of the following: calculation of an activation function in a neural network, and data type conversion.

5. The device according to claim 1, wherein the data cache module is configured to: According to the description information about the output data in the acquired context information, a corresponding transformation operation is performed on the received data to obtain the output data, and the transformed data and the corresponding instruction information are cached.

6. The apparatus according to claim 5, wherein the transformation operation comprises a combination of one or more of the following: rotation, inversion, transposition, merging, and splitting.

7. The apparatus according to claim 1, wherein the control module is further configured to: Receive status information related to the instruction, and set or adjust context information of the corresponding instruction in the context cache according to the received status information.

8. According to the device described in any one of claims 1-7, the corresponding instruction information transmitted along the pipeline together with the data in the data path includes an instruction identifier, and each module in the data path is configured to identify the corresponding instruction from the instruction cache according to the instruction identifier, and to identify context information related to the instruction from the context cache.

9. According to the device described in any one of claims 1-7, the corresponding instruction information transmitted along the pipeline together with the data in the data path includes an instruction identifier and a context identifier, and each module in the data path is configured to identify the corresponding instruction from the instruction cache according to the instruction identifier, and to identify context information related to the instruction from the context cache according to the context identifier.

10. The apparatus according to claim 9, wherein the control module is further configured to: An association relationship is established between the instruction to be processed stored in the instruction cache and the relevant context information of the instruction stored in the context cache by setting a mapping between the instruction identifier and the context identifier, and each module in the data path is configured to identify context information related to the currently processed instruction from the context cache according to the association relationship.

11. A processor, comprising the multi-threaded data transfer device according to any one of claims 1 to 10, which is used for transferring data between an on-chip cache and an off-chip memory of the processor.

Images