Data handling device for parallel processor and corresponding processor

Abstract

The present application provides a data handling device supporting parallel processors, which can be used for data transfer between on-chip cache and off-chip memory. The device includes a control module, an arithmetic logic unit and its associated aligned cache, and a data path running in a pipeline manner consisting of a read-in DMA module, a data cache module and an output DMA module. The arithmetic logic unit used for performing calculation operations such as precision conversion is pre-placed at the data interface of the data storage unit of the processor, so that the data output from the data storage unit is first subjected to precision conversion by the arithmetic logic unit, and then transmitted to the data path, thereby minimizing the line width between the data storage module and the data handling module on the processor chip, thereby saving chip power consumption and area.

Description

Technical Field

[0001] The present application relates to a parallel processor, and more particularly to data transfer between different memories in a parallel processor. Background Art

[0002] The statements in this section are merely intended to provide background information related to the technical solution of the present application to aid understanding, and they do not necessarily constitute prior art for the technical solution of the present application.

[0003] Artificial intelligence (AI) technology has developed rapidly in recent years and has penetrated into various fields such as visual perception, speech recognition, assisted driving, smart home, traffic dispatch, etc. Many AI algorithms involve learning and computing based on neural networks, such as convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), etc. These AI algorithms require strong parallel computing capabilities to process massive data, so processors that support multi-core parallel computing such as GPUs and GPGPUs are usually used to perform multi-layer neural network operations. These parallel processors are also generally referred to as neural network accelerators. These neural network accelerators usually complete the calculation work of each layer in the order of the layers of the neural network and use the calculation results as the input of the next layer or the next network until the final output result is obtained. For the calculation work of each layer, the neural network accelerator will divide the corresponding calculation work into multiple tasks according to the maximum task size supported by each layer itself. The output of each task will be temporarily stored in the memory (such as on-chip cache or off-chip memory, etc.). After all the tasks of each layer are completed, the outputs of each task will be merged in the memory to obtain the calculation results of the corresponding layer.

[0004] Such processors usually include internal data storage units to store the output of each task calculation. However, since the amount of data processed by neural networks is usually large, and the internal storage unit space of the processor is limited, it is often necessary to move the calculation results in the internal data storage unit to the on-chip cache or off-chip memory for subsequent merging to obtain the calculation results of the current layer. How to efficiently move large amounts of data between various levels of memory is an urgent problem to be solved in neural network accelerators.

[0005] It should be noted that the above content is only used to help understand the technical solution of the present application, and is not used as a basis for evaluating the prior art of the present application. Summary of the invention

[0006] The inventors have also found in practice that in addition to the need to move a large amount of data between different neural networks or between different layers of neural networks, the requirements for the accuracy of the processed data are often different. Therefore, the transformation of data accuracy is also one of the tasks that must be completed in the neural network accelerator.

[0007] Therefore, in an embodiment of the present application, a data handling device suitable for a neural network accelerator is provided, which can simultaneously implement a method for data precision conversion during the data handling process, thereby improving the computing efficiency of the neural network accelerator.

[0008] According to the first aspect of the embodiment of the present application, a data handling device for a parallel processor is provided, comprising a control module, a data path running in a pipeline manner, an arithmetic logic unit, and an alignment cache associated with the arithmetic logic unit, wherein the data path comprises a read-in DMA module, a data cache module, and an output DMA module, wherein one side of the arithmetic logic unit is directly coupled to the data interface of the data storage unit of the parallel processor, and the other side is communicatively connected to the read-in DMA module. The read-in DMA module can send a data request to the data storage unit according to the control information from the control module, and save the configuration information corresponding to the data request to the alignment cache associated with the arithmetic logic unit, and pass the received data to the next module in the data path. The arithmetic logic unit is used to receive the data returned by the data storage unit for the data request, and perform a preset processing operation on the received data according to the configuration information corresponding to the data request obtained from the alignment cache, and transmit the processed data to the read-in DMA module.

[0009] In the data handling device of this embodiment, the arithmetic logic unit ALU used to perform calculation operations such as precision conversion is pre-placed at the data interface of the data storage unit, so that the data output from the data storage unit is first subjected to precision conversion and then transmitted. This can minimize the connection line width between the data storage module and the data handling module, thereby further saving chip power consumption and area.

[0010] In some embodiments, the data handling device may also include a request pipeline module and a data pipeline module, wherein one side of the request pipeline module is coupled to the configuration interface of the data storage unit and connected to the alignment cache, and the other side is connected to the read-in DMA module through a data connection, and one side of the data pipeline module is connected to the arithmetic logic unit, and the other side is connected to the read-in DMA module through a data connection. The request pipeline module is used to pipeline beat the data request received from the read-in DMA module and its corresponding configuration information, and pass the pipelined data request to the data storage unit, and save the pipelined configuration information to the alignment cache. The data pipeline module is used to pipeline beat the data received from the arithmetic logic unit, and pass the pipelined data to the read-in DMA module. In some embodiments, a MUX module may also be provided between the arithmetic logic unit and the data pipeline module, which is used to rearrange the data received from the arithmetic logic unit according to the configuration information obtained from the alignment cache, and transmit the rearranged data to the data pipeline module.

[0011] In the data handling device of this embodiment, two pipeline beat modules are introduced to improve the timing and synchronization of each module inside the data handling module, and further avoid the problem of data processing timing disorder inside the data handling module caused by data connection transmission delay between data storage modules and data handling modules in different partitions of the chip.

[0012] In some embodiments, the preset processing operation performed by the arithmetic logic unit may include a combination of one or more of the following: calculation of activation functions in a neural network, data precision conversion, and quantization operations. In some embodiments, the aligned cache may be a first-in-first-out cache queue. In some embodiments, the configuration information corresponding to the data request may be generated by a control module based on received context information related to the instruction.

[0013] In some embodiments, the control module may also include an instruction cache and a context cache, wherein the context cache is shared by each module in the data path. The control module is also used to store the received instructions to be processed in the instruction cache, and store the context information related to the instructions to be processed in the context cache. The read-in DMA module is also used to obtain the instructions to be processed from the instruction cache and generate a data request and its corresponding configuration information according to the context information related to the instruction obtained from the context cache, and pass the received data together with the corresponding instruction information to the next module in the data path, wherein the corresponding instruction information is used to identify the current instruction and its related context information. The data cache module is also used to select the corresponding context information from the context cache according to the corresponding instruction information received, cache the received data and the corresponding instruction information according to the obtained context information, and pass the cached data and its corresponding instruction information together to the next module in the data path when the cached data meets the output requirements. The output DMA module is also used to select the corresponding context information from the context cache according to the corresponding instruction information received, generate an output address according to the obtained context information, and output the output address together with the received data.

[0014] In this embodiment, instructions and contexts are processed separately, and the context cache module is shared by modules in a data path operating in pipeline mode. Data and instructions are transmitted in the pipeline, but the context is not transmitted. This improves the parallelism of multiple instruction executions while reducing chip area consumption.

[0015] In some embodiments, the read-in DMA module is further configured to: while passing the received data and corresponding instruction information to the next module in the data path, feedback instruction completion information to the control module and obtain the next instruction to be processed from the instruction cache; the output DMA module is further configured to: while outputting the output address and the received data, feedback instruction completion information to the control module; the control module is further configured to: clear the completed instruction from the instruction cache in response to the instruction completion information from the read-in DMA module, and clear the context information related to the completed instruction from the context cache in response to the instruction completion information from the output DMA module. In the data handling device, the data path running in a pipeline manner is adopted, and data and corresponding instruction information are transmitted along the pipeline but context information is not transmitted, thereby improving the data handling efficiency while minimizing the area overhead.

[0016] In some embodiments, the data cache module can be configured to: perform corresponding transformation operations on the received data to obtain output data according to the description information about the output data in the acquired context information, and cache the transformed data and corresponding instruction information, wherein the transformation operation includes one or more of the following combinations: rotation, inversion, transposition, merging, and splitting. In this way, the conversion of input and output data is performed in advance during the data handling process, reducing the computing load of the computing core, which is particularly suitable for neural networks.

[0017] According to a second aspect of the embodiment of the present application, a parallel processor is provided, comprising the data handling device according to the first aspect of the embodiment of the present application, which is used to transfer data between the on-chip cache and the off-chip memory of the parallel processor. Such a processor is particularly suitable for neural network computing.

[0018] The technical solution of the embodiment of the present application may have the following beneficial effects:

[0019] A data handling device suitable for a neural network accelerator is provided, which can realize a method of data precision conversion during data handling, thereby improving the computing efficiency of the accelerator. In addition, the arithmetic logic unit used for precision calculation is pre-placed to the data interface of the storage unit, which can minimize the connection line width between the data storage module and the data handling module, thereby achieving the purpose of saving chip power consumption and area.

[0020] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0022] Figure 1 A schematic diagram of a structural module of a data handling device according to an embodiment of the present application.

[0023] Figure 2 This is a schematic diagram of the structural modules of a data handling device according to another embodiment of the present application.

[0024] Figure 3 This is a schematic diagram of the structural modules of a data handling device according to another embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of contradictions or conflicts, the various embodiments provided in the present application can be combined with each other, and some or all of the features in the various embodiments can be used in combination with each other.

[0026] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0029] The inventors found in practice that although the access speed of data can be improved by setting up an on-chip cache inside the chip to pre-fetch part of the data and cache the calculation results, the capacity of the on-chip cache is usually limited due to limitations such as cost and volume. Especially in applications involving neural networks, such limited on-chip caches often cannot cache all data related to multiple layers of the neural network. In addition, the inventors also tried to use direct memory access (DMA) to move data between the on-chip cache and the off-chip memory, but found that each time DMA is enabled, each computing core needs to start and configure DMA transmission separately, which greatly reduces the execution efficiency of a large number of multi-threaded tasks on multiple cores. Although it is possible to try to improve the parallelism between instructions through the DMA of the pipeline architecture, in SIMT processors used to execute AI algorithms such as neural networks, in addition to instructions, additional contexts are often required to describe the data to be processed (such as the data arrangement of the input and output of each layer of the neural network, data type, feature map size, etc.), if these contexts are propagated along the pipeline together with the instructions, it will inevitably cause a large area consumption.

[0030] In an embodiment of the present application, a data handling device is provided, which separates instruction information from context information related to the instruction. This can not only simplify the instructions while minimizing area overhead, but also facilitate flexible control of the context information of the instructions. It is particularly suitable for parallel processors that support multiple threads.

[0031] Figure 1 A functional module diagram of a data handling device according to an embodiment of the present application is given. The data handling module includes: a control module and a data path running in a pipeline manner consisting of a read-in DMA module, a data cache module and an output DMA module. The control module of the data handling module receives instructions and status information issued by the processor, and reads a section of data from the data storage unit of the processor according to the instructions and status information and writes it to another memory. The instructions received here refer to instructions that require the participation of the data handling device, which may include but are not limited to data handling instructions such as store instructions (STORE) or load instructions (LOAD). The data storage unit here may be but is not limited to a register array, an on-chip random access memory (which may be referred to as an on-chip RAM), etc. The control module may also instruct the output DMA to output data to the next-level data storage unit. The other memory here may be but is not limited to a register array, an on-chip RAM, a cache memory, a memory, an external memory, and any other data storage device.

[0032] In this embodiment, the control module also includes an instruction cache and a context cache. The control module stores the received one or more instructions to be processed and their related context information in the instruction cache and the context cache respectively. Among them, the context information related to the instruction can generally refer to all relevant information required to complete the instruction, for example, it can include: information related to the execution of the instruction, information related to the data processed by the instruction, etc. Different instructions have different context information, and this document does not impose any restrictions on this. The processor can send these context information related to the instruction as status information or configuration information together with the instruction to the data handling device. In an embodiment of the present application, the instruction cache and the context cache can be shared by each module in the data path.

[0033] When processing an instruction, each module in the data path can extract context information related to the instruction from the context cache; after completing the processing of an instruction, the data involved in the instruction and the corresponding instruction information can be transmitted to the next module in the data path, and the next instruction can be obtained for processing. The corresponding instruction information transmitted in the data path together with the data here is: information required for each level of modules in the data path to identify the corresponding instruction to be processed and its context (for example, instruction identifier, context identifier, etc.), and does not include specific context information related to the instruction. In this data handling device, the data path running in a pipeline manner transmits data and corresponding instruction information along the pipeline but does not transmit context information, thereby improving the data handling efficiency while minimizing the area overhead.

[0034] More specifically, refer to Figure 1 When the read-in DMA module receives an instruction from the control module or extracts an instruction to be processed from the instruction cache, the context information corresponding to the instruction is found from the context cache module. The read-in DMA module can generate a data request to an external storage module based on the context information related to the instruction obtained. The data request at least includes the address and size of the data to be read. After receiving the data request, the external storage module returns the data to the read-in DMA module. The read-in DMA module can output the received data together with the relevant information of the currently processed instruction (for example, the instruction identifier) ​​to the next module in the data path, such as the data cache module. After the read-in DMA module completes the transmission, the read-in DMA module can immediately feedback the instruction completion information to the control module, and obtain the next instruction from the instruction cache, and at the same time obtain the context information of the instruction in the context cache module, so that the operation of the next instruction can be started.

[0035] Optionally, after receiving the instruction completion information from the read-in DMA module, the control module can clear the completed instruction from the instruction cache to prepare for receiving the next instruction. However, the context information related to the completed instruction stored in the context cache module still needs to be retained for use by subsequent modules in the pipeline. In some embodiments, since the volume of the instruction itself is very small (the volume of the instruction information used to indicate the transported data is small), compared to the method of transmitting the instruction itself and the specific context information corresponding to the instruction along the pipeline, in the case where the pipelined data path does not directly transmit the specific context information, even if the instruction itself is transmitted along the pipeline, it will not cause a large area overhead. Each module in the data path can choose to transmit the current instruction itself and / or the instruction identifier of the current instruction along the pipeline, but the pipelined data path still does not directly transmit the context information of the instruction, but each module in the data path obtains the context information required by each module from the shared context cache according to the corresponding instruction information received and the current instruction and / or instruction identifier. Among them, if the pipeline does not pass the specific context corresponding to the instruction, nor does it directly pass the instruction itself, but instead transfers data by passing an identifier and obtaining the cache content based on the identifier, the instructions can be streamlined and the bandwidth overhead occupied during data transfer can be further reduced.

[0036] The data cache module can be used to receive data and corresponding instruction information output from the upper module in the data path (for example, it can be a read-in DMA module). The data cache module can select corresponding context information from the context cache according to the corresponding instruction information received, and cache the received data and corresponding instruction information according to the acquired context information. When the data meets the output requirements, the data cache module can pass the cached data and its corresponding instruction information to the next level module in the data path (for example, the output DMA module). For example, the data cache module can start outputting after detecting that there is enough data in the cache, or it can start outputting when the amount of cached data can meet the basic data requirements of the output channel. For another example, the data cache module can merge the data according to the relevant information in the context information to increase the bandwidth utilization of the output data.

[0037] In some embodiments, the data cache module can also perform corresponding transformation operations on the received data to obtain corresponding output data according to the description information about the output data in the acquired context information, and cache the transformed data and the corresponding instruction information. The transformation operation here may include but is not limited to one or more combinations of the following: rotation, reversal, transposition, merging, splitting, etc. For example, the data cache module can perform transformation operations such as rotation, reversal and transposition of different dimensions on the received data according to the description information such as the input data arrangement information and the output data arrangement information contained in the context information, so that the input data can better adapt to the format of the output data. This is particularly suitable for the processing of multi-layer neural networks, because the input data dimensions, formats or types required by different network layers may be different, and the data handling module will have higher adaptability when performing transformation operations in the process of caching data. When the processor finishes processing a network layer, the processing result will be written into the cache and saved to the memory through the data handling module, and when starting the processing of the next network layer, the data can be directly read without performing data format conversion. This is because the data handling module has completed the corresponding data conversion and adaptation work in the process of saving data. This obviously reduces the computing load of the processor and is conducive to improving the throughput of the processor for large data.

[0038] Continue to refer Figure 1 The output DMA module receives data and corresponding instruction information output from the upper-level module (e.g., data cache module) in the data path, and selects context information associated with the instruction from the context cache according to the received corresponding instruction information. The output DMA module can generate an output address according to the acquired context information, and output the output address together with the received data, for example, writing the received data to the corresponding address in the target memory. At the same time, the output DMA module can feedback instruction completion information to the control module. The control module clears the context information related to the instruction from the context cache in response to the instruction completion information from the output DMA module. The control module can also send a notification to an external related control unit to indicate that the data handling device is ready to receive the next instruction.

[0039] In some embodiments, in the data path of the data handling device, an arithmetic logic unit module (which may be referred to as an ALU module) located before the data cache module may also be included. The ALU module selects context information related to the instruction from the context cache according to the corresponding instruction information received from the read-in DMA module, and performs a preset operation on the data received from the read-in DMA module according to the context information obtained. Such preset operations include but are not limited to: calculation of activation functions in neural networks, data precision conversion, quantization, etc. The ALU module passes the processed data and the corresponding instruction information to the next module in the data path (for example, it can be passed to the data cache module). It should be understood that the ALU module here is an optional module, which is particularly suitable for neural network processing. For example, in a multi-layer neural network, the output feature data of the previous layer is provided to the next layer as the input feature data of the next layer for further processing. Each layer often operates the input neuron vector and the corresponding weight vector, and then the obtained operation result is passed through the activation function to generate the output feature. Such an activation function will be repeated in each layer. Therefore, in the embodiment of the present application, an ALU module is provided in the data handling device to process the activation function, so that the execution of these repeated activation functions is independent of each computing core. In this way, after processing the calculation of one layer of the neural network, the calculation results can be directly saved in the corresponding on-chip cache. The data handling device can complete the execution of the activation function during the handling of these intermediate results and save the data processed by the activation function; and when starting the next layer of the neural network, the computing core can directly obtain the saved data for calculation. Through such an ALU module, some repeated fixed operations can be performed, reducing the computing load of the computing core and improving the data processing throughput of the processor.

[0040] In the above-mentioned embodiment, the data path of the data handling device operates in a pipeline manner. Each module in the data path transmits data and corresponding instruction information along the pipeline, so each module can immediately obtain the next instruction for processing after processing one instruction, thereby improving the parallelism of instructions. In addition, in the data handling device, part of the instruction information such as the instruction identifier is transmitted along the pipeline in the data path together with the data, and there is no need to transmit all the context information related to the instruction in the pipeline, thereby improving the data handling efficiency and reducing the chip area overhead.

[0041] Furthermore, the inventors found in their research and implementation of processor chips that in actual processor chips, the modules responsible for data storage and the modules responsible for data handling are usually placed in different partitions of the chip, and there is a long data connection between the two modules. However, the amount of data that needs to be transmitted in neural network operations is very large. In order to improve efficiency, the data connection between the module responsible for data storage and the module responsible for data handling in the processor chip is usually relatively wide. These connections occupy a large chip area, and a large amount of data transmission will also cause a large power loss. Therefore, in some other embodiments, a data handling device with a structure different from that of the above embodiments is provided, which pre-places the arithmetic logic unit ALU used for calculation operations such as precision conversion to the data interface of the data storage unit, so that the data output from the data storage unit is first subjected to precision conversion, and then transmitted, thereby minimizing the line width between the data storage module and the data handling module, thereby further saving chip power consumption and area.

[0042] Figure 2 A schematic diagram of the structural modules of a data handling device according to some other embodiments of the present invention is given. Figure 1 The data handling device shown has a different structure. Figure 2 In the embodiment, the arithmetic logic unit ALU and its associated aligned cache are pre-placed at the data interface of the data storage unit in the processor, and the data output from the data storage unit is first processed by the ALU before being passed to the read-in DMA module. One side of the arithmetic logic unit is directly coupled to the data interface of the data storage unit of the parallel processor, and the other side is communicatively connected to the read-in DMA module, for example, it can be directly connected to the read-in DMA module through a data connection, or it can be indirectly connected to the read-in DMA module through other intermediate modules.

[0043] More specifically, the control module of the data handling module receives the instructions and status information sent by the processor, and reads a piece of data from the data storage unit of the processor according to the instructions and status information and writes it to another memory. As mentioned above, the read-in DMA module will send data request information to the data storage module according to the control information from the control module. The data request information may include a data request enable signal, a data address, etc. The data address is calculated by the read-in DMA module according to the context information related to the instruction contained in the status information received by the control module. In this embodiment, the read-in DMA module will save the data configuration information required for the arithmetic logic unit ALU to perform the operation in the aligned cache associated with the ALU while sending the data request. The data configuration information also comes from the context information related to the instruction contained in the status information received by the control module, for example, it may include the source data format, the target data format, and the activation function type. In this embodiment, each data request sent by the read-in DMA module corresponds to a data configuration information, and for each data request, the data storage unit receiving the data request will output a corresponding data.

[0044] like Figure 2 As shown, the data storage unit returns data for each data request, which is first output to the arithmetic logic unit ALU set at the data interface of the data storage unit. The ALU module receives the data returned by the data storage unit for each data request, obtains the data configuration information corresponding to each data from the alignment cache to perform corresponding operations, and passes the results of the operations to the next level module, such as the read-in DMA module. The operations performed by the arithmetic logic unit ALU can be data precision conversion (such as conversion from FP32 to FP16, conversion from Float to Int, conversion from FP32 to BF16, etc.), quantization, and calculation of activation functions (such as ReLU, Tanh, SigMod, etc.). Data precision conversion generally includes conversion from low-precision type to high-precision type and conversion from high-precision type to low-precision type. Quantization generally refers to the process of mapping the input value in a large set (usually a continuous set) to a smaller set (usually with a limited number of elements). For example, in the field of machine learning, this quantization often refers to the conversion of floating-point numbers with a relatively wide bit width into fixed-point numbers with a relatively low bit width (such as FP32->INT8, FP32->INT4, etc.). It should be understood that data precision conversion, quantization, and calculation of activation functions are common operations in the field of neural networks. This article is only used as an example and does not impose any limitation.

[0045] In a typical chip hardware design, each module on the chip has a preset maximum supported data width, and the bit width of the data connection between the modules is also designed according to the maximum supported data width. Therefore, the data connection between the data storage unit and the data handling device is also set according to the larger value of the data widths supported by the two. Usually, the data width of the data returned by the data storage unit is usually larger than the maximum data width supported by the data handling module. Assuming that the maximum supported data width of the data handling module is B bits, and the maximum supported data width of the data storage unit is n*B bits, where n is a natural number greater than 1, if data with a width of n*B bits is transmitted between the two modules, the data connection bit width requires n*B bits. As mentioned above, in an actual processor chip, the module responsible for data storage and the module responsible for data handling are usually placed in different partitions of the chip. If there is a longer data connection with a higher bit width between the two modules, it will take up a large chip area and require a longer transmission delay. In this embodiment, by Figure 2 As shown, the arithmetic logic unit ALU and its associated alignment cache are placed in front of the data interface of the data storage unit, so that the bit width of the data connection between the data storage module and the data handling module is only B bits, which can effectively reduce the routing width between different chip partitions, the layout area of ​​the chip and the power consumption. The maximum precision conversion operation supported by the ALU module is set according to the maximum data width supported by the data storage unit and the maximum data width supported by the data handling device.

[0046] The inventors have also found in practice that when data is transmitted along the data connection between the data storage module and the data handling module located in different partitions of the chip, the data transmission time often exceeds one clock cycle. Such data transmission delay often causes the data processing timing disorder inside the data handling module, especially for the processing timing of each module in the data path running in a pipeline manner in the data handling module. Therefore, in some embodiments of the present invention, two pipeline beat modules are introduced to improve the timing and synchronization of each module inside the data handling module.

[0047] Figure 3 A schematic diagram of the structural modules of a data handling device according to some other embodiments of the present invention is given. Figure 1 The control module introduced in this paper and the data path running in a pipelined manner consisting of the read DMA module, the data buffer module and the output DMA module, as well as the above combined Figure 2In addition to the arithmetic logic unit ALU and its associated alignment cache arranged at the data interface of the data storage unit of the processor, the data handling device also includes a request pipeline module, a data pipeline module and a MUX module arranged at the data interface of the data storage unit of the processor. One side of the request pipeline module is coupled to the configuration interface of the data storage unit and is also connected to the alignment cache, and the other side is connected to the read-in DMA module through a data connection. One side of the data pipeline module is connected to the arithmetic logic unit, and the other side is connected to the read-in DMA module through a data connection.

[0048] More specifically, the request pipeline module receives the data request and data configuration information sent from the read-in DMA module of the data transfer device, and simultaneously performs pipeline beats for a certain number of levels on the two information, passes the pipelined data request information to the data storage unit, and transmits the data configuration information to the alignment cache module.

[0049] The alignment cache module receives the data configuration information delayed by the request pipeline module and caches the data configuration information. The alignment cache module can be implemented in the form of a first-in-first-out queue, for example. The ALU module and the multiplexer (MUX) module in the data handling device can obtain the corresponding data configuration information from the alignment cache according to the requirements to perform the corresponding operation.

[0050] The data storage unit stores the data results calculated by the processor, and returns the corresponding data according to the data request transmitted by the data pipeline module. The returned data is first transmitted to the ALU module connected to the data interface of the data storage unit. When the ALU module receives the data returned from the data storage unit for the corresponding data request, it obtains the data configuration information corresponding to the data request from the alignment cache to perform the corresponding operation, and transmits the result of the operation to the next-level MUX module. On the chip, the same data interface often reuses data of different data types, and the arrangement of different data is different. For example, the number of FP16 type data that can be transmitted by the same 128Byte data interface is 64, while the number of INT8 type data transmitted is 128. The multiplexer MUX module can realize the reuse of data of different data types for the same data interface. When the MUX module receives the data output from the ALU module, it rearranges the received data according to the corresponding data configuration information in the alignment cache, and transmits the processed data to the data pipeline module. The width of the input and output data of the MUX module is B (bit), which is the same as the input and output data width of the data handling device. It should be understood that the MUX module is optional and its position is variable. In some embodiments, the MUX module can also be arranged in the data path before the data cache module.

[0051] The data pipeline module receives data from the upper-level module (such as the ALU module or the MUX module), beats the data through a certain number of pipeline levels, and then passes the beaten data to the read-in DMA module.

[0052] After receiving the return data from the data pipeline, the read DMA module passes it to the next module in the pipeline, such as the data cache module. The read DMA includes N+1 data channels, and the data width of each data channel is B / N (bit), where N+1 represents the number of storage blocks (banks) in the data cache module.

[0053] The data cache module can write the data information transmitted by the DMA into the internal cache in a specific order according to the status information received by the control module, and read the data in the cache in a specific order and pass it to the output DMA module. These modules will not change the width of each data, that is, the data width is B (bit). As mentioned above, the data cache module can also perform functions such as input and output rate matching of the data handling device and dimensional transformation of input and output data.

[0054] The output DMA module writes the data transmitted by the data cache module to the external memory in a certain format according to the status information received by the control module. The external memory here can be a storage resource such as cache, on-chip RAM, register array and internal memory. The output DMA module can generate a write control information for each data according to the status information received by the control module. Each write control information can include but is not limited to write enable, write address and write mask. At the same time, the output DMA module can also divide the output data into M channels according to the specifications of the external memory to be written, and the data width of each channel is B (bit) / M, where M is the number of storage blocks (banks) of the external memory.

[0055] In this embodiment, both the request pipeline and the data pipeline are used to beat the pipeline so that the timing of each module in the pipeline is synchronized. For example, if the operation time of the previous stage is exactly equal to the operation time of the next stage, the output of the previous stage can be directly merged into the input of the next stage; if the operation time of the previous stage is greater than the operation time of the next stage, the output data of the previous stage needs to be properly cached before it can be merged into the input of the next stage; if the operation time of the previous stage is less than the operation time of the next stage, the data needs to be cached at the previous stage to avoid overflow of the next stage data. Therefore, the request pipeline and the data pipeline usually include a group of registers, each register beats a beat (equivalent to inserting a pipeline stage), which is used to cache and delay the data transmitted on the pipeline so that the timing of the previous module and the next module is synchronized. In this embodiment, the number of registers contained in the request pipeline and the data pipeline (i.e., the number of pipeline beats or levels that need to be inserted) can be determined in the chip design stage according to the distance of the chip layout between the data storage module and the data handling module. The longer the distance between the data storage module and the data handling module on the chip is, the longer the time delay caused by data transmission is, and accordingly, the more registers are included in the request pipeline and the data pipeline.

[0056] In some other embodiments of the present application, a parallel processor suitable for a neural network is also provided, which includes the above-mentioned Figure 1 , Figure 2 or Figure 3 The data handling device introduced is used to execute data transfer between different memories of the parallel processor.

[0057] It should be understood that although the working principle of the data handling module machine is introduced above using a neural network accelerator, such a data handling module is not limited to a neural network accelerator, but can also be applied to parallel processors such as GPGPU, GPU and other general-purpose vector processors.

[0058] It should be understood that for the modules such as read-in DMA module, read-out DMA module, ALU module, control module or controller mentioned in this article and the functions they perform, in addition to being implemented in a computer-readable program, it is entirely possible to implement the same functions by logically programming the corresponding functional modules, processes or steps so that these modules are in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, the functional module implemented in this way can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as the internal structure of the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules that implement related processes or method steps and structures within the hardware component.

[0059] References in this specification to "various embodiments," "some embodiments," "one embodiment," or "an embodiment," etc. refer to a particular feature, structure, or property described in conjunction with the embodiment being included in at least one embodiment. Thus, the appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., in various places throughout the specification do not necessarily refer to the same embodiment. Furthermore, particular features, structures, or properties may be combined in any suitable manner in one or more embodiments. Thus, particular features, structures, or properties shown or described in conjunction with one embodiment may be combined in whole or in part with features, structures, or properties of one or more other embodiments without restriction, as long as the combination is not illogical or inoperable.

[0060] The expressions of "including", "having" and similar meanings in this specification are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices. "One" or "an" does not exclude multiple situations. In addition, the elements in the drawings of this application are for illustrative purposes only and are not drawn to scale.

[0061] Although the present application has been described through the above-mentioned embodiments, the present application is not limited to the embodiments described herein, and also includes various changes and modifications made without departing from the scope of the present application.

Claims

1. A data handling device for a parallel processor, comprising a control module, a data path running in a pipeline manner, an arithmetic logic unit, and an alignment cache associated with the arithmetic logic unit, wherein the data path comprises a read-in DMA module, a data cache module, and an output DMA module, wherein one side of the arithmetic logic unit is directly coupled to a data interface of a data storage unit of the parallel processor, and the other side is communicatively connected to the read-in DMA module; and among them: The read-in DMA module is used to send a data request to the data storage unit according to the control information from the control module, save the configuration information corresponding to the data request to the alignment buffer associated with the arithmetic logic unit, and pass the received data to the next module in the data path; The arithmetic logic unit is used to receive the data returned by the data storage unit in response to the data request, and to perform a preset processing operation on the received data according to the configuration information corresponding to the data request obtained from the alignment cache, and to transmit the processed data to the read-in DMA module.

2. The apparatus according to claim 1, further comprising a request pipeline module and a data pipeline module, wherein one side of the request pipeline module is coupled to the configuration interface of the data storage unit and connected to the alignment cache, and the other side is connected to the read-in DMA module through a data connection, and one side of the data pipeline module is connected to the arithmetic logic unit, and the other side is connected to the read-in DMA module through a data connection; and wherein The request pipeline module is used to pipeline the data request received from the read DMA module and its corresponding configuration information, and transmit the pipelined data request to the data storage unit, and save the pipelined configuration information to the alignment cache; The data pipeline module is used to perform pipeline beats on the data received from the arithmetic logic unit, and pass the pipeline beat data to the read-in DMA module.

3. According to the device according to claim 2, a MUX module is also arranged between the arithmetic logic unit and the data pipeline module, which is used to rearrange the data received from the arithmetic logic unit according to the configuration information obtained from the alignment cache, and transmit the rearranged data to the data pipeline module.

4. According to the device described in any one of claims 1-3, the preset processing operations performed by the arithmetic logic unit include a combination of one or more of the following: calculation of activation functions in a neural network, data precision transformation, and quantization operations. The device according to claim 4 , wherein the aligned cache is a first-in-first-out cache queue.

6. The device according to any one of claims 1-3, wherein the configuration information corresponding to the data request is generated by a control module according to received context information related to the instruction.

7. The apparatus according to claim 6, wherein the control module further comprises an instruction cache and a context cache, wherein the context cache is shared by each module in the data path, and wherein: The control module is further used to store the received instructions to be processed in the instruction cache, and store the context information related to the instructions to be processed in the context cache; The read-in DMA module is further used to obtain the instruction to be processed from the instruction cache and generate a data request and its corresponding configuration information according to the context information related to the instruction obtained from the context cache, and pass the received data together with the corresponding instruction information to the next module in the data path, wherein the corresponding instruction information is used to identify the current instruction and its related context information; The data cache module is used to select corresponding context information from the context cache according to the received corresponding instruction information, cache the received data and the corresponding instruction information according to the obtained context information, and pass the cached data and its corresponding instruction information to the next module in the data path when the cached data meets the output requirements; The output DMA module is used to select corresponding context information from the context cache according to the received corresponding instruction information, generate an output address according to the obtained context information, and output the output address together with the received data.

8. The device according to claim 7, wherein: The read DMA module is further configured to: while passing the received data together with the corresponding instruction information to the next module in the data path, feed back instruction completion information to the control module and obtain the next instruction to be processed from the instruction cache; The output DMA module is further configured to: feed back instruction completion information to the control module while outputting the output address together with the received data; The control module is further configured to: clear completed instructions from the instruction cache in response to instruction completion information from the read DMA module, and clear context information associated with the completed instructions from the context cache in response to instruction completion information from the output DMA module.

9. The device according to claim 1, wherein the data cache module is configured to: According to the description information about the output data in the acquired context information, a corresponding transformation operation is performed on the received data to obtain the output data, and the transformed data and the corresponding instruction information are cached, wherein the transformation operation includes a combination of one or more of the following: rotation, reversal, transposition, merging, and splitting.

10. A parallel processor, comprising the data handling device according to any one of claims 1 to 9, which is used for transferring data between an on-chip cache and an off-chip memory of the parallel processor.

Images