Tensor data exchange circuit, data stream processing device and method

Abstract

The invention provides a tensor data exchange circuit and a data stream processing device and method.The tensor data exchange circuit is used for a data stream processor and comprises a first non-blocking full-arrangement assembly and a second non-blocking full-arrangement assembly; performing data format conversion of in-block data exchange and inter-block data exchange on a plurality of sub-data blocks split by input data through the first non-blocking full-arrangement component and the second non-blocking full-arrangement component so as to output data meeting the format requirement of a target component; wherein the first non-blocking full-arrangement component is used for carrying out in-block data exchange on the sub-data blocks; and the second non-blocking full-arrangement component performs inter-block data exchange on the plurality of sub-data blocks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data flow processing, and in particular to a tensor data exchange circuit, a data flow processing device and method. BACKGROUND

[0002] In a data flow processor, dynamic arrangement of tensor data (such as filtering, copying, exchanging, rearranging and transposing) is a core requirement for efficient computing. With the increasing complexity of artificial intelligence algorithms and the diversification of scenarios, the processor needs to dynamically adapt to multiple data patterns to meet the input requirements of different functional modules. Traditional solutions have significant limitations: 1. Customized circuit modules (Prior Art 1): relying on fixed function circuits to implement specific data exchange patterns, lacking dynamic reconfiguration capability, making it difficult to adapt to rapid algorithm iteration and multi-scenario requirements. 2. Full permutation network (Prior Art 2): Based on Benes network and other non-blocking exchange architectures, it supports arbitrary input-output mapping, but its hardware resource consumption is huge. For example, a 16x16 matrix exchange requires a 256x256 scale Benes network, consuming 1920 2x2 switch units, resulting in significant increase in chip area and power consumption (see CN109445752B).

[0003] Although Benes network achieves non-blocking full permutation through multi-level interconnection and cross-connection, its high number of switch units limits its application in resource-sensitive scenarios. Therefore, existing technologies cannot balance dynamic reconfigurability, low hardware overhead and high energy efficiency, resulting in performance bottlenecks for artificial intelligence processors in complex models and diversified scenarios. The industry urgently needs an exchange solution that balances dynamic reconfigurability and hardware efficiency to support flexible data arrangement while reducing resource overhead, thereby improving the computing efficiency and practicality of artificial intelligence processors. SUMMARY

[0004] The present application aims to provide a tensor data exchange circuit, a data flow processing device and method to support multiple modes of filtering, copying, exchanging, rearranging and transposing of input data.

[0005] To achieve the above purpose, an embodiment of the present application provides a tensor data exchange circuit for a data flow processor, which can include a first non-blocking full permutation component and a second non-blocking full permutation component; the first non-blocking full permutation component and the second non-blocking full permutation component perform block-in data exchange and block-to-block data exchange data format conversion on multiple sub-data blocks split from input data to output data that meets the format requirements of a target component; wherein the first non-blocking full permutation component performs block-in data exchange on the sub-data blocks; the second non-blocking full permutation component performs block-to-block data exchange on multiple sub-data blocks.

[0006] In one embodiment, the data format conversion performed by the switching circuit through the first non-blocking permutation component and the second non-blocking permutation component includes at least one of filtering, copying, swapping, rearranging, and transposing.

[0007] In one embodiment, the first non-blocking permutation component includes a plurality of first non-blocking permutation networks corresponding to the sub-data blocks one-to-one.

[0008] In one embodiment, the first non-blocking permutation component performs the same or different intra-block data swapping on part or all of the input sub-data blocks through the first non-blocking permutation networks.

[0009] In one embodiment, the second non-blocking permutation component includes a second non-blocking permutation network, and the number of input ports of the second non-blocking permutation network is greater than or equal to the number of the first non-blocking permutation networks.

[0010] In one embodiment, the bit width of a single input port of the second non-blocking permutation network is greater than or equal to the total output bit width of a single first non-blocking permutation network.

[0011] In one embodiment, the second non-blocking permutation component includes a plurality of second non-blocking permutation networks, and the second non-blocking permutation networks form a cascaded structure, the first layer of the second non-blocking permutation networks is connected to the first non-blocking permutation component, and the number of input ports of the next layer of the second non-blocking permutation networks is greater than or equal to the number of the previous layer of the second non-blocking permutation networks.

[0012] In one embodiment, in the cascaded structure, the bit width of a single input port of the next layer of the second non-blocking permutation networks is greater than or equal to the total output bit width of a single second non-blocking permutation network of the previous layer.

[0013] In one embodiment, the number of layers of the cascaded structure is greater than or equal to the number of times of splitting of the input data.

[0014] In one embodiment, the second non-blocking permutation component performs the same or different inter-block data swapping on part or all of the input sub-data blocks through each layer of the second non-blocking permutation networks.

[0015] An embodiment of the present application further provides a data stream processing device comprising a tensor data exchange circuit, the device further comprising a control unit; the control unit adjusts the transmission path from input to output of the non-blocking full permutation network in the first non-blocking full permutation component and the second non-blocking full permutation component according to the data format conversion requirement of the component connected with the tensor data exchange circuit through data exchange control instruction, so that the tensor data exchange circuit completes data format conversion through intra-block data exchange and inter-block data exchange.

[0016] In an embodiment, the control unit further comprises generating corresponding control instructions according to the data format conversion requirement and the data source component of the data format to be converted, and controlling the data source component to split the data of the data format to be converted into a plurality of sub-data blocks and then input into the tensor data exchange circuit through the control instructions.

[0017] In an embodiment, the data source component comprises a combination of any two or more of a communication engine, a storage unit, a vector and scalar operation unit, and a matrix operation unit; the tensor data exchange circuit performs data format conversion on the data transmitted between any two of the communication engine, the storage unit, the vector and scalar operation unit, and the matrix operation unit according to the data exchange control instruction.

[0018] An embodiment of the present application further provides a data stream processing method, the method comprising: generating corresponding control instructions according to the data format conversion requirement of the component connected with the tensor data exchange circuit through data exchange control instruction, and according to the data source component of the data format to be converted; adjusting the transmission path from input to output of the non-blocking full permutation network in the first non-blocking full permutation component and the second non-blocking full permutation component in the tensor data exchange circuit through the data exchange control instruction; controlling the data source component to split the data of the data format to be converted into a plurality of sub-data blocks and then input into the tensor data exchange circuit through the control instructions, so that the tensor data exchange circuit completes corresponding data format conversion through intra-block data exchange and inter-block data exchange.

[0019] In an embodiment, generating corresponding control instructions according to the data format conversion requirement and the data source component of the data format to be converted comprises: determining a data splitting strategy according to the storage type of the data of the data format to be converted in the data source component and the data format conversion requirement; and generating corresponding control instructions according to the data splitting strategy.

[0020] In an embodiment, the data splitting strategy comprises splitting according to specified axis granularity and splitting according to block granularity; wherein the splitting according to specified axis granularity comprises splitting according to row granularity direction, splitting according to column granularity direction, and splitting according to channel direction granularity of three-dimensional data.

[0021] An embodiment of the present application further provides a chip comprising the data flow processing device.

[0022] An embodiment of the present application further provides a board card comprising the chip.

[0023] An embodiment of the present application further provides an electronic device comprising the board card or the chip.

[0024] The tensor data exchange circuit, the data flow processing device and the method provided by the present application can significantly improve the dynamic arrangement capability and resource efficiency of an artificial intelligence processor through an innovative block strategy and a reconfigurable network design, and the specific technical advantages include hardware resource efficiency, dynamic multi-mode compatibility, high throughput and low power consumption collaborative optimization, and flexible scalable architecture. Through the exchange mechanism of the "block-global" cooperation, the inherent contradiction between the functional flexibility and the hardware efficiency of the traditional scheme is broken, and the core data arrangement capability of the artificial intelligence processor with high energy efficiency, low delay and strong adaptability is provided for the data flow architecture, which helps the efficient deployment and real-time inference of complex models.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory, but not limiting the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings described herein are used to provide further understanding of the present application, form a part of the present application, and do not constitute a limitation of the present application. In the drawings:

[0027] Figure 1A A schematic diagram of the principle structure of the tensor data exchange circuit provided by an embodiment of the present application;

[0028] Figure 1B A schematic diagram of the benes network provided by an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the application structure of the tensor data exchange circuit provided by an embodiment of the present application;

[0030] Figure 3 A schematic diagram of the data splitting and conversion process under the cascade structure provided by an embodiment of the present application;

[0031] Figure 4 A schematic diagram of the structure of the data flow processing device provided by an embodiment of the present application;

[0032] Figure 5 A schematic diagram of the flow of the data flow processing method provided by an embodiment of the present application;

[0033] Figure 6 This is a schematic diagram of the control instruction generation process provided in an embodiment of this application;

[0034] Figure 7A and Figure 7B This is a schematic diagram of matrix data splitting logic at different granularities provided in an embodiment of this application;

[0035] Figure 8A and Figure 8B This is a schematic diagram of matrix data transformation logic under row and column granularity splitting logic provided in an embodiment of this application;

[0036] Figures 9A to 9C This is a schematic diagram of matrix data conversion logic under block-level splitting logic provided in an embodiment of this application;

[0037] Figure 10 This is a schematic diagram of matrix data transposition logic under block-level splitting logic provided in an embodiment of this application;

[0038] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0039] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0040] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0041] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0042] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0043] Please refer toFigure 1A As shown, an embodiment of this application provides a tensor data exchange circuit for a data stream processor, the exchange circuit including a first non-blocking full permutation component and a second non-blocking full permutation component.

[0044] The first non-blocking full permutation component and the second non-blocking full permutation component are used to perform intra-block data exchange and inter-block data exchange on multiple sub-data blocks split from the input data, so as to output data that meets the format requirements of the target component.

[0045] Wherein, the first non-blocking permutation component performs intra-block data exchange on the sub-data blocks; the second non-blocking permutation component performs inter-block data exchange on multiple sub-data blocks; the data format conversion includes at least one of filtering, copying, exchanging, rearranging, and transposing.

[0046] In the above embodiments, the first non-blocking full permutation component may include multiple first non-blocking full permutation networks corresponding one-to-one with the sub-data blocks; the first non-blocking full permutation component may include at least one second non-blocking full permutation network. It should be noted that the first non-blocking full permutation network and the second non-blocking full permutation network have the same non-blocking full permutation network structure; different names are used for ease of understanding, but this does not mean that they need to have different structures; please refer to... Figure 1B As shown, the non-blocking full permutation network can be a strictly non-blocking network such as a Benes network or a Clos network, or a generalized non-blocking network. The connection relationships between the input and output ports of these non-blocking full permutation networks are not unique; therefore, intermediate-level crossovers can be adjusted accordingly based on network structure transformation information, thereby changing the input and output transmission paths to achieve, for example... Figure 1B The sequential input A shown 0，0 To A 3，3The output order can vary, resulting in different effects. The target component refers to a subsequent processing or storage unit connected to the tensor data exchange circuit and having specific data format requirements. The target component may include, but is not limited to, a communication engine, storage unit, vector and scalar operation unit, matrix operation unit, etc. For example, in a data stream processing device, a matrix operation unit typically requires input data to be arranged in a two-dimensional matrix format, either row-wise or block-wise; while a communication engine may require data to be arranged in a channel or specific interleaving format when sending data; and a storage unit may require data to be arranged in a block-interleaving manner to maximize bandwidth utilization when writing data. The tensor data exchange circuit, through the collaborative work of the first non-blocking full permutation component and the second non-blocking full permutation component, splits the input data into multiple sub-data blocks, performs intra-block and inter-block data exchange respectively, and finally outputs data that meets the format requirements of the target component, thereby achieving efficient and seamless data flow between different processing units. The format conversion logic based on the target component will be described in detail in subsequent embodiments and will not be elaborated here.

[0047] To reduce the number of switching units used, this application employs a two-stage non-blocking full-permutation component combined with a block-based strategy to achieve data format conversion. Specifically, taking a real-world scenario as an example, for m×m tensor data, a total of m... 2 If only a non-blocking full permutation network (such as a Benes network) is used to process these m data points... 2 To exchange data, both the input and output nodes need to be m. 2 Based on the calculation formula for a non-blocking full permutation network, the required number of 2×2 switching units is: When m is large, the number of switching units required becomes enormous; for example, with m=16, the total number of switching units in a non-blocking permutation network is 1920, which would consume a large amount of logic resources if implemented in this way. To ensure the correct implementation of matrix data exchange and minimize the number of switching units, this application splits the original m×m tensor data into m sub-data blocks, then inputs these sub-data blocks into m sets of Benes networks to complete intra-block data exchange and outputs m sets of vector data of length m. The output vector data of length m is then treated as a whole and passed through a Benes network with m input ports to achieve inter-block data exchange, finally outputting the exchanged m×m data. From a performance perspective, for m×m tensor data, the consumption of 2×2 switching units in this scheme is... Taking m=16 as an example, the total number of switching units consumed is 952; compared with the existing method, the consumption of 2×2 switching units can be reduced by half.

[0048] Specifically, in the above embodiments, to achieve efficient inter-block data exchange, this application employs a key data encapsulation and transmission mechanism: the entire vector of length m output by each first non-blocking full permutation network is treated as a complete data block. At the hardware level, the physical bit width of a single input port of the second non-blocking full permutation network responsible for inter-block exchange is configured to be greater than or equal to the total bit width of such a data block (i.e., the sum of the bit widths of m data elements). This allows each data block to be treated as an atomic unit, and within one transmission cycle, it can be entirely fed into the network through one input port of the second non-blocking full permutation network. Please refer to... Figure 2 As shown, in the above embodiments, the intra-block data exchange and inter-block data exchange mainly rely on the characteristic that the connection relationship in the non-blocking permutation network can be changed and adjusted. For intra-block data exchange, each sub-data block is used as input data. For example, for a sub-data block, its data format is an m×m one-dimensional vector. At this time, it can be input into a first non-blocking permutation network with m input ports according to the order of each element in the one-dimensional vector. Then, by changing the internal data transmission path of the first non-blocking permutation network, the mapping relationship from the input port to the output port can be reconstructed, thereby completing the data conversion to generate a new one-dimensional vector and realizing the change of the element relationship within the block.

[0049] For inter-block data exchange, multiple sub-data blocks correspond to multiple one-dimensional vectors. Each sub-data block's corresponding one-dimensional vector is treated as a single element and input to an input port in the second non-blocking permutation network. Therefore, n sub-data blocks require the second non-blocking permutation network to have n input ports. By changing the internal data transmission path of this second non-blocking permutation network, the relationship between the n sub-data blocks can be altered. It should be noted that intra-block data exchange and inter-block data exchange do not have a specific execution order. When the data flow passes through the first non-blocking permutation component first and then the second non-blocking permutation component, intra-block data exchange occurs first, followed by inter-block data exchange; conversely, when the data flow passes through the second non-blocking permutation component first and then the first non-blocking permutation component, inter-block data exchange occurs first, followed by intra-block data exchange. This satisfies more data format conversion requirements. The following sections will provide a detailed explanation of intra-block and inter-block data exchange using actual data format conversion examples; further details will not be provided here.

[0050] In one embodiment of this application, the first non-blocking permutation component performs intra-block data exchange on some or all of the input sub-data blocks through the first non-blocking permutation network, either with the same or different data.

[0051] Specifically, in the first non-blocking full permutation component, not all of the first non-blocking full permutation networks need to be used to process sub-data blocks. For example, if there are m first non-blocking full permutation networks in the first non-blocking full permutation component, but only m-1 sub-data blocks, then only m-1 first non-blocking full permutation networks are used to perform the corresponding intra-block data exchange. Overall, the number of first non-blocking full permutation networks needs to be greater than or equal to the number of sub-data blocks after splitting. Secondly, each first non-blocking full permutation network is an independent entity, and there is no consistency requirement between the first non-blocking full permutation networks. During the above intra-block data exchange process, any two first non-blocking full permutation networks can perform the same or different intra-block data exchange logic. For example, the first non-blocking full permutation network numbered A performs intra-block data exchange of type X, and the first non-blocking full permutation network numbered B performs intra-block data exchange of type Y, where X and Y belong to different exchange logics, that is, the transmission paths from the input port to the output port within the two are not the same. Thus, more granular data exchange control is achieved, which is not only more flexible, but also more adaptable to different data format conversion scenarios.

[0052] In one embodiment of this application, the second non-blocking full permutation component includes a second non-blocking full permutation network, wherein the number of input ports of the second non-blocking full permutation network is greater than or equal to the number of the first non-blocking full permutation networks. Further, the bit width of a single input port of the second non-blocking full permutation network is greater than or equal to the total output bit width of a single first non-blocking full permutation network, so as to transmit all output data of any first non-blocking full permutation network as a complete data block.

[0053] Specifically, given that each input port of the second non-blocking full permutation network corresponds to one of the first non-blocking full permutation networks, so as to perform inter-block data exchange on the sub-data blocks input by the non-blocking full permutation networks after intra-block data exchange; for this purpose, the number of input ports of the second non-blocking full permutation network must be greater than or equal to the total number of the first non-blocking full permutation networks, so that all output elements of the first non-blocking full permutation networks can be input to the second non-blocking full permutation network.

[0054] In one embodiment of this application, the second non-blocking full permutation component includes multiple second non-blocking full permutation networks. These second non-blocking full permutation networks form a cascaded structure. A first-layer second non-blocking full permutation network is connected to the first non-blocking full permutation component, and the number of input ports of a subsequent-layer second non-blocking full permutation network is greater than or equal to the number of input ports of the preceding-layer second non-blocking full permutation network. Further, in the cascaded structure, the bit width of a single input port of a subsequent-layer second non-blocking full permutation network is greater than or equal to the total output bit width of a single input port of the preceding-layer second non-blocking full permutation network, so as to transmit all output data of any preceding-layer second non-blocking full permutation network as a complete data block.

[0055] For details, please refer to Figure 3 As shown, in this embodiment, the second non-blocking permutation network can be determined according to the number of times the input data is split. For example, if data A, after being split once to generate A1, A2, and A3, still needs further splitting to complete the fine processing, it can be further split into A11, A12, A13, A21, A22, A23, A31, A32, and A33. At this time, A11, A12, A13, A21, A22, A23, A31, A32, and A33 each correspond to one of the first non-blocking permutation networks. The three first non-blocking permutation networks responsible for converting A11, A12, and A13 correspond to the first second non-blocking permutation network of the first layer to perform data format conversion to generate B1. The three first non-blocking permutation networks of A21, A22, and A23 correspond to the second second non-blocking permutation network of the first layer to perform data format conversion to generate B2. The three first non-blocking permutation networks of A31, A32, and A33... A non-blocking full permutation network corresponds to the third second non-blocking full permutation network in the first layer to perform data format conversion to generate B3. Subsequently, the second non-blocking full permutation network forming the first layer cascade structure connects B1, B2, and B3 to the second non-blocking full permutation network in the second layer cascade structure. The second non-blocking full permutation network in the second layer cascade structure generates B through inter-block data exchange, thereby realizing the process of generating data B from data A after data format conversion. It should be noted that in the above embodiment, the number of layers of the cascade structure is greater than or equal to the number of times the input data is split. For example, if the input data is split twice, the cascade structure has two layers. Similarly, if the input data is split N times, the cascade structure has N layers. Those skilled in the art can further set N plus m layers, where m is an integer greater than or equal to 1. In this way, through redundant design, it can be applied to more data format conversion scenarios and avoid the problem of hardware incompatibility due to too many times of input data splitting.

[0056] In practical applications, the aforementioned cascaded structure primarily addresses the need for multi-level splitting of input data. For example, a 100×100 matrix can be split into 16 25×25 matrices, and each 25×25 matrix can be further split into 25 5×5 matrices. Accordingly, the cascaded structure must contain at least two layers. Subsequently, 25 first non-blocking full permutation networks are used to exchange data between the sub-data blocks of the 25 5×5 matrices, and the output is sent to the second non-blocking full permutation network of the first layer for inter-block data exchange. Then, the output data of the second non-blocking full permutation network of the first layer is sent to the second non-blocking full permutation network of the second layer, where inter-block data exchange is performed, thereby completing the data format conversion of the 100×100 matrix.

[0057] In the above embodiments, the number of input ports of the second non-blocking permutation network in the next layer is greater than or equal to the number of the second non-blocking permutation networks in the previous layer. The principle is the same as the logic that the number of input ports of the second non-blocking permutation network is greater than or equal to the number of the first non-blocking permutation network, and will not be described in detail here.

[0058] In one embodiment of this application, the second non-blocking permutation component performs the same or different inter-block data exchange on some or all of the input sub-data blocks through each layer of the second non-blocking permutation network.

[0059] Specifically, in the second non-blocking full permutation component, not all of the multiple second non-blocking full permutation networks need to be used to process sub-data blocks. For example, if there are m second non-blocking full permutation networks in the second non-blocking full permutation component, but only m-1 sub-data blocks, then only m-1 second non-blocking full permutation networks are used to perform the corresponding inter-block data exchange. Overall, the number of second non-blocking full permutation networks needs to be greater than or equal to the number of split sub-data blocks. Secondly, each second non-blocking full permutation network is an independent entity, and there is no consistency requirement between the second non-blocking full permutation networks. During the above-mentioned inter-block data exchange process, any two second non-blocking full permutation networks can perform the same or different intra-block data exchange logic. For example, the second non-blocking full permutation network numbered A performs inter-block data exchange of type X, and the second non-blocking full permutation network numbered B performs inter-block data exchange of type Y, where X and Y belong to different exchange logics, that is, the transmission paths from the input port to the output port within the two are not the same. Thus, more granular data exchange control is achieved, which is not only more flexible, but also more adaptable to different data format conversion scenarios.

[0060] In summary, the tensor data exchange circuit provided in this application significantly improves the dynamic orchestration capability and resource efficiency of artificial intelligence processors through innovative block strategy and reconfigurable network design. Specific technical advantages include high hardware resource efficiency, dynamic multi-mode compatibility, high throughput and low power consumption co-optimization, and flexible and scalable architecture.

[0061] In one embodiment of this application, a data stream processing apparatus including the tensor data exchange circuit is also provided. The apparatus further includes a control unit. The control unit adjusts the input-to-output transmission path of the non-blocking permutation network in the first non-blocking permutation component and the second non-blocking permutation component through data exchange control commands according to the data format conversion requirements of the components connected to the tensor data exchange circuit, so that the tensor data exchange circuit completes the data format conversion through intra-block data exchange and inter-block data exchange.

[0062] In the above embodiments, when the data format conversion requirement is tensor data to matrix data, the input-to-output transmission path of the non-blocking permutation network within the first and second non-blocking permutation components can be adjusted to control the tensor data exchange circuit to perform functions such as filtering / copying and transposing, thereby completing the conversion from tensor data to matrix data or vice versa. In practice, the data source component may include any combination of two or more of the following: a communication engine, a storage unit, a vector and scalar operation unit, and a matrix operation unit. The tensor data exchange circuit performs data format conversion on the data transmitted between any two of the communication engine, the storage unit, the vector and scalar operation unit, and the matrix operation unit according to the data exchange control command.

[0063] For details, please refer to Figure 4 As shown, in practical operation, the tensor data exchange circuit needs to convert the tensor data received by the communication engine from other processor nodes into the data format required by the matrix operation unit; or convert the matrix data output by the matrix operation unit into tensor data, so that the communication engine can send it to other processor nodes. For this data stream, the control unit controls the tensor data exchange circuit to perform functions such as filtering / copying and transposing through control commands.

[0064] In another embodiment, the tensor data exchange circuit needs to convert the matrix data output by the matrix operation unit into vector / scalar data for processing by the vector / scalar processing unit; or shape the output result of the vector / scalar operation unit into matrix data format for subsequent processing by the matrix operation unit. For this data stream, the control unit controls the tensor data exchange circuit to perform functions such as filtering / copying and rearranging through control commands.

[0065] In another embodiment, the tensor data exchange circuit needs to convert the tensor data read in parallel from the storage unit into the data format required by the vector / scalar operation unit; or convert the matrix data output by the vector / scalar operation unit into tensor data in a specific interleaving format, so as to facilitate parallel writing to different blocks of the storage unit with maximum bandwidth. For this data stream, the control unit controls the tensor data exchange circuit to perform functions such as filtering / copying and exchanging through control commands.

[0066] In another embodiment, the tensor data exchange circuit needs to convert tensor data read in parallel from the storage unit into tensor data required by the communication engine, so that the communication engine can send it to other processor nodes; or convert tensor data received by the communication engine from other processor nodes into tensor data in a specific interleaving format, so that it can be written in parallel to different blocks of the storage unit with maximum bandwidth. For this data stream, the control unit controls the tensor data exchange circuit to perform functions such as switching and rearranging through control commands.

[0067] In another embodiment, the tensor data exchange circuit needs to convert the tensor data read in parallel from the storage unit into the data format required by the matrix operation unit; or convert the matrix data output by the matrix operation unit into tensor data in a specific interleaving format, so as to facilitate parallel writing to different blocks of the storage unit with maximum bandwidth. For this data stream, the control unit controls the tensor data exchange circuit through control commands to mainly realize functions such as exchange, rearrangement, and transpose.

[0068] In another embodiment, the tensor data exchange circuit needs to convert the tensor data received by the communication engine from other processor nodes into the data format required by the vector / scalar operation unit; or convert the data output by the vector / scalar operation unit into tensor data, so that the communication engine can send it to other processor nodes. For this data stream, the control unit controls the tensor data exchange circuit to perform functions such as filtering / copying and exchanging through control commands.

[0069] It should be noted that this application requires the input data to be split before performing data format conversion. Specifically, the control unit further includes generating corresponding control instructions based on the data format conversion requirements and the data source component of the data format to be converted. These control instructions control the data source component to split the data into multiple sub-data blocks before inputting them into the tensor data exchange circuit. This splitting process is mainly implemented by the control unit controlling the data source component. The specific splitting logic will be described in detail in subsequent embodiments and will not be elaborated here.

[0070] Therefore, this application can support the mapping of input tensor data to output tensor data in multiple modes through the above-mentioned data stream processing device, which significantly improves the dynamic orchestration capability and resource efficiency of artificial intelligence processors; it not only has a significant improvement in hardware resource efficiency and dynamic multi-mode compatibility, but also achieves high throughput and low power consumption collaborative optimization and flexible scalable architecture.

[0071] Please refer to Figure 5 As shown, in one embodiment of this application, a data stream processing method is also provided, the method comprising:

[0072] S501 generates corresponding control commands based on the data format conversion requirements of the components connected to the tensor data exchange circuit and the data source components of the data format to be converted.

[0073] S502 adjusts the input-to-output transmission path of the non-blocking permutation network in the first and second non-blocking permutation components of the tensor data exchange circuit through the data exchange control command.

[0074] S503 controls the data source component to split the data to be converted into multiple sub-data blocks and input them into the tensor data exchange circuit through the control command, so that the corresponding data format conversion can be completed by the tensor data exchange circuit using intra-block data exchange and inter-block data exchange.

[0075] The data format conversion includes at least one of filtering, copying, exchanging, rearranging, and transposing.

[0076] In practice, the execution entity of the above data stream processing method is the aforementioned control unit. The control commands in this method can be obtained through the following process; please refer to [reference needed]. Figure 6 As shown, the generation of corresponding control instructions based on the data format conversion requirements and the data source component of the data format to be converted includes:

[0077] S601 determines a data splitting strategy based on the storage type of the data to be converted in the data source component and the data format conversion requirements;

[0078] S602 generates corresponding control instructions based on the data splitting strategy.

[0079] The data splitting strategy includes splitting by a specified axis granularity and splitting by block granularity; wherein, splitting by a specified axis granularity includes splitting by row granularity, splitting by column granularity, and splitting by channel granularity of three-dimensional data.

[0080] In this embodiment, the intensity splitting according to the channel direction of the three-dimensional data can be carried out in the following way:

[0081] For m×m×m three-dimensional tensor data, one-dimensional expansion can be performed in the following two ways: Method 1: Decompose into m×m two-dimensional tensors at the granularity of rows (X), columns (Y), or the third axis (Z), and then decompose them into one-dimensional vectors using the two-dimensional tensor decomposition method. Method 2: Decompose into sub-three-dimensional tensor data at the granularity of blocks. For example, the m×m×m three-dimensional tensor data can be decomposed into m sub-vectors of size... Each sub-tensor block is further expanded into m groups of one-dimensional vectors of length m in a row-, column-, or third-axis-first manner. Similarly, higher-dimensional tensor data can also be processed using a similar scheme, which will not be detailed here.

[0082] To facilitate a clearer understanding of the data splitting logic provided in this application, the following uses a two-dimensional tensor as an example to illustrate the above data splitting strategy. Those skilled in the art will understand that this splitting strategy is only for the purpose of understanding the above data splitting logic and does not impose any limitations on it.

[0083] For m×m matrix data (two-dimensional tensor), one-dimensional expansion can be performed in two ways. Method 1: Split into one-dimensional vectors at the row or column level; an m×m matrix can be split into m groups of one-dimensional vectors of length m. Method 2: Split into one-dimensional vectors at the block level, dividing the m×m matrix data into m blocks of size m. The submatrix blocks are expanded into m groups of one-dimensional vectors of length m in either row-major or column-major order. For example... Figure 7A and Figure 7B This demonstrates two ways to expand the original 4×4 matrix data, where row / column granularity ( Figure 7A This expands the original matrix data into four groups of vectors, either row-wise or column-wise; block granularity ( Figure 7B The original matrix data is expanded into 4 groups of vectors according to 2×2 sub-blocks.

[0084] By controlling the expansion of matrix data into vectors, tensor data can be filtered, copied, exchanged, rearranged, and transposed. In practical applications, the expansion method can be dynamically configured through software programming. In method 1, data filtering, copying, and exchanging are implemented row-wise or column-wise based on the original matrix data, and rearrangement of the original matrix data at the row or column level is also possible. In method 2, data filtering, copying, and exchanging are implemented based on sub-data blocks within the original matrix data, allowing rearrangement of the original matrix data at the sub-data block level, and transposition of the original matrix data.

[0085] Taking an m×m matrix (two-dimensional tensor) as an example, this section explains the principles of filtering, copying, exchanging, rearranging, and transposing tensor data. The subscripts i and j represent the row and column numbers of the matrix, respectively, and the letter A represents a matrix element. Therefore, a specific element in the matrix can be represented as A. i，j ;

[0086] The filtering operation refers to setting certain elements in the tensor data to 0 according to a certain pattern, which is to sparse the tensor.

[0087] Assume the filter pattern is: :

[0088] Filtering operations can only be performed at the block level. The filtering operations are as follows:

[0089] ;

[0090] Therefore, the filtering pattern is used to perform the corresponding filtering operation on the two-dimensional tensor data. During this process, A i0,j0 Represents the first data in the first row and first column, and similarly, A in,j0 For the data in the nth row and 1st column, A in,jn This refers to the data in the nth row and nth column. A copy operation refers to copying certain elements, rows, or columns from a tensor data set.

[0091] Single element copy:

[0092] ;

[0093] Copy by row: Copy a row of data into a two-dimensional tensor.

[0094] ;

[0095] Copy by column: Copy a column of data into a two-dimensional tensor.

[0096] ;

[0097] Supports data exchange operations between any tensor elements within a block:

[0098] ;

[0099] Tensor data rearrangement operations, such as rearranging by row or by column:

[0100] ;

[0101] ;

[0102] The tensor data transpose operation results in a matrix shape of m×m after transpose.

[0103] ;

[0104] Based on the above principles of filtering, copying, exchanging, rearranging, and transposing, please refer to the example of a 4×4 matrix. Figure 8A As shown, the process of filtering and copying matrix data using method 1 is as follows:

[0105] When using data filtering: By filtering the first row, element A in the first row of the matrix is ​​removed. 0，1 A 0，2 and A 0，3 Filter to 0, and keep the original value of other elements.

[0106] When copying elements: Copy element A from the first row of the matrix. 0，1 Copy to another location on this line.

[0107] When performing row copying: the elements in row 0 are copied and filled into rows 1, 2, and 3.

[0108] When copying columns: the elements in column 3 are copied and filled into columns 0, 1, and 2.

[0109] Please refer to this again. Figure 8B As shown, the process for swapping and rearranging functions is as follows:

[0110] When exchanging data: Element A in row 0, column 1. 0，1 With element A in row 0 and column 3 0，3 To exchange;

[0111] When performing row rearrangement: swap all elements in row 0 with all elements in row 2;

[0112] When rearranging columns: swap all elements in column 1 with all elements in column 3;

[0113] Therefore, according to Figure 8A and Figure 8B The method shown can realize the functions of filtering, copying, exchanging and rearranging matrix data; in actual work, these functions can be used in combination or individually, and this application does not impose any restrictions.

[0114] There are some differences between Method 2 and Method 1 when implementing matrix data filtering, copying, swapping, rearranging, and transposing. Please refer to [link / reference needed] for details. Figure 9A As shown, the process of filtering and copying matrix data using method 2 is as follows:

[0115] When performing data filtering: Element A in sub-block 0 of the matrix. 1，0 and A 0，1 Filter to 0, and keep the original value of other elements.

[0116] When copying elements: Element A of sub-block 0 in the matrix will be copied. 0，1 Copy to other locations within this sub-block;

[0117] When performing block copying: the elements of child block 0 are copied and filled into child blocks 1, 2, and 3.

[0118] Please refer to this again. Figure 9B As shown, the data exchange and block rearrangement process is as follows:

[0119] Data exchange: Transfer element A from sub-block 0 0，1 With element A 1，0 To exchange;

[0120] Block rearrangement: Swap all elements of sub-block 0 with all elements of sub-block 3 in their corresponding positions.

[0121] Please refer to further information. Figure 9C As shown, the transpose operation process is as follows:

[0122] Transpose operation: According to the rules of matrix transpose, all elements in a 4×4 matrix are swapped and rearranged to obtain the transposed matrix. For specific implementation details, please refer to [link / reference]. Figure 10 As shown, the 4×4 matrix data is first split into 4 sub-blocks with a granularity of 2×2. Then, the Benes network is used to perform intra-block transpose operation. The output result is then passed through a first-level Benes network to perform inter-block rearrangement, thus obtaining the transpose result of the original 4×4 matrix data.

[0123] The dataflow processing method provided in this application significantly improves the dynamic orchestration capabilities and resource efficiency of artificial intelligence processors through an innovative block-based strategy and reconfigurable network design. By employing a "block-global" collaborative exchange mechanism, it overcomes the inherent contradiction between functional flexibility and hardware efficiency in traditional solutions, providing dataflow-based artificial intelligence processors with core data orchestration capabilities that combine high energy efficiency, low latency, and strong adaptability, thus facilitating the efficient deployment and real-time inference of complex models.

[0124] In one embodiment of this application, a chip is also provided, the chip including the aforementioned data stream processing device.

[0125] In one embodiment of this application, a board card is also provided, which includes the aforementioned chip.

[0126] Since the chip and the board are similar in principle to the data stream processing method in solving the problem, the implementation of the chip and the board can refer to the implementation of the data stream processing method, and the repeated parts will not be described again.

[0127] In one embodiment of this application, an electronic device is also provided, which has the aforementioned board or chip deployed thereon.

[0128] Figure 11 This is a block diagram illustrating an electronic device 900 for implementing the above-described data processing method, according to an exemplary embodiment. For example, the electronic device 900 may be an AI server, a training and promotion integrated machine, etc.

[0129] Reference Figure 11 The electronic device 900 may include one or more of the following components: an AI-accelerated computing module, a CPU module, a power supply module, a hard drive module, and a fan module. Each module works in conjunction with the bus system through a standardized hardware interface, with the specific architecture as follows:

[0130] The AI-accelerated computing module comprises multiple AI accelerator cards deployed in parallel. Each AI accelerator card integrates at least one AI accelerator chip (such as an RPU chip, GPU chip, or CGRA chip). Data communication between the AI ​​accelerator cards is achieved through a high-speed card-to-card (C2C) interconnect structure, supporting low-latency, high-bandwidth horizontal scaling. The AI ​​accelerator chip is dedicated to performing AI computing tasks such as high-density matrix operations, neural network model training, and / or inference, providing the main computing power support.

[0131] The CPU module includes at least one CPU board, which houses a central processing unit (CPU) chip and associated CPU memory (such as DDR4 / DDR5, RAM). The CPU chip serves as the system control center, responsible for task scheduling, resource allocation, I / O management, and coordinating the parallel computing of the AI ​​acceleration computing module, while also handling non-accelerated general-purpose computing tasks.

[0132] The power module is equipped with redundant power supply units to provide stable power distribution and management for the AI ​​acceleration computing module, CPU module and other modules.

[0133] The hard drive module integrates a high-speed solid-state drive (SSD) and / or a large-capacity hard disk drive (HDD), connected to the system bus via a backplane. The hard drive stores the operating system, AI training datasets, model parameters, and computation results, providing high-throughput data read / write channels and supporting data preprocessing and persistence.

[0134] The fan module uses a multi-zone independent speed-controlled fan array, which is configured in key heat source areas (such as AI accelerator cards and CPU heat dissipation areas) to achieve system heat dissipation through forced air cooling and ensure the stable operation of high-efficiency computing components.

[0135] The CPU module is connected to the AI ​​acceleration computing module via the PCIe bus to enable task distribution, result collection, and memory coordination.

[0136] The CPU module manages the data access of the hard drive module through SATA / SAS / NVMe interfaces.

[0137] The power module provides tiered power to all functional modules through the power distribution backplane.

[0138] The fan module adjusts the fan speed based on temperature monitoring signals from the CPU board and AI accelerator card.

[0139] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0140] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0141] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A tensor data exchange circuit for a data stream processor, characterized in that, The switching circuit includes a first non-blocking full-arrangement component and a second non-blocking full-arrangement component; The first non-blocking full permutation component and the second non-blocking full permutation component are used to perform intra-block data exchange and inter-block data exchange on multiple sub-data blocks split from the input data, so as to output data that meets the format requirements of the target component. The first non-blocking permutation component performs intra-block data exchange on the sub-data blocks; the second non-blocking permutation component performs inter-block data exchange on multiple sub-data blocks.

2. The tensor data exchange circuit according to claim 1, characterized in that, The data format conversion performed by the switching circuit through the first non-blocking permutation component and the second non-blocking permutation component includes at least one of filtering, copying, swapping, rearranging, and transposing.

3. The tensor data exchange circuit according to claim 1, characterized in that, The first non-blocking permutation component includes multiple first non-blocking permutation networks that correspond one-to-one with the sub-data blocks.

4. The tensor data exchange circuit according to claim 3, characterized in that, The first non-blocking permutation component performs intra-block data exchange on some or all of the input sub-data blocks through the first non-blocking permutation network, either with the same or different data.

5. The tensor data exchange circuit according to claim 3, characterized in that, The second non-blocking permutation component includes a second non-blocking permutation network, wherein the number of input ports of the second non-blocking permutation network is greater than or equal to the number of the first non-blocking permutation networks, and the bit width of a single input port of the second non-blocking permutation network is greater than or equal to the total output bit width of a single first non-blocking permutation network.

6. The tensor data exchange circuit according to claim 1, characterized in that, The second non-blocking full permutation component includes multiple second non-blocking full permutation networks, which form a cascaded structure. The first layer of the second non-blocking full permutation network is connected to the first non-blocking full permutation component, and the number of input ports of the second non-blocking full permutation network in the next layer is greater than or equal to the number of the second non-blocking full permutation networks in the previous layer. In the cascaded structure, the bit width of a single input port of the second non-blocking full permutation network in the next layer is greater than or equal to the total output bit width of a single second non-blocking full permutation network in the previous layer.

7. The tensor data exchange circuit according to claim 6, characterized in that, The number of layers in the cascaded structure is greater than or equal to the number of times the input data is split.

8. The tensor data exchange circuit according to claim 6, characterized in that, The second non-blocking permutation component performs the same or different inter-block data exchange on some or all of the input sub-data blocks through each layer of the second non-blocking permutation network.

9. A data stream processing apparatus comprising a tensor data exchange circuit according to any one of claims 1 to 8, characterized in that, The device also includes a control unit; The control unit adjusts the input-to-output transmission path of the non-blocking permutation network in the first and second non-blocking permutation components according to the data format conversion requirements of the components connected to the tensor data exchange circuit, so that the tensor data exchange circuit can complete the data format conversion through intra-block data exchange and inter-block data exchange.

10. The data stream processing apparatus according to claim 9, characterized in that, The control unit also includes generating corresponding control instructions based on the data format conversion requirements and the data source component of the data format to be converted. The control instructions control the data source component to split the data of the format to be converted into multiple sub-data blocks and then input them into the tensor data exchange circuit.

11. The data stream processing apparatus according to claim 10, characterized in that, The data source component includes any combination of two or more of the following: a communication engine, a storage unit, a vector and scalar operation unit, and a matrix operation unit. The tensor data exchange circuit performs data format conversion on any two of the communication engine, the storage unit, the vector and scalar operation unit, and the matrix operation unit according to the data exchange control command.

12. A data stream processing method, characterized in that, The method includes: Based on the data format conversion requirements of the components connected to the tensor data exchange circuit, the corresponding control instructions are generated by the data format conversion requirements and the data source components of the data format to be converted. The data exchange control command adjusts the input-to-output transmission path of the non-blocking permutation network in the first and second non-blocking permutation components of the tensor data exchange circuit. The control command controls the data source component to split the data to be converted into multiple sub-data blocks and input them into the tensor data exchange circuit, so that the tensor data exchange circuit can complete the corresponding data format conversion by using intra-block data exchange and inter-block data exchange.

13. The data stream processing method according to claim 12, characterized in that, The generation of corresponding control instructions based on the data format conversion requirements and the data source component of the data format to be converted includes: The data splitting strategy is determined based on the storage type of the data to be converted in the data source component and the data format conversion requirements; The corresponding control instructions are generated based on the data splitting strategy.

14. The data stream processing method according to claim 13, characterized in that, The data splitting strategy includes splitting by specified axis granularity and splitting by block granularity; The splitting by specified axis granularity includes splitting by row granularity, splitting by column granularity, and splitting by channel granularity of three-dimensional data.

15. A chip, characterized in that, The chip includes the data stream processing apparatus according to any one of claims 9 to 11.

16. A circuit board, characterized in that, Includes the chip as described in claim 15.

17. An electronic device, characterized in that, It is equipped with the board as described in claim 16 or the chip as described in claim 15.

Images