Tensor data exchange circuit, data flow processing apparatus and method

Abstract

The application provides a tensor data exchange circuit, a data stream processing device and a method. The tensor data exchange circuit is used for a data stream processor and comprises a first non-blocking full permutation component and a second non-blocking full permutation component. The first non-blocking full permutation component and the second non-blocking full permutation component are used for performing data format conversion on a plurality of sub-data blocks split from input data, performing intra-block data exchange and inter-block data exchange, and outputting data meeting target component format requirements. The first non-blocking full permutation component performs intra-block data exchange on the sub-data blocks. The second non-blocking full permutation component performs inter-block data exchange on the plurality of sub-data blocks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data flow processing, and in particular to a tensor data exchange circuit, a data flow processing device and method. BACKGROUND

[0002] In a data flow processor, dynamic arrangement of tensor data (such as filtering, copying, exchanging, rearranging and transposing) is a core requirement for efficient computing. With the increasing complexity of artificial intelligence algorithms and the diversification of scenarios, the processor needs to dynamically adapt to multiple data patterns to meet the input requirements of different functional modules. Traditional solutions have significant limitations: 1. Customized circuit modules (Prior Art 1): relying on fixed function circuits to implement specific data exchange patterns, lacking dynamic reconfiguration capability, making it difficult to adapt to rapid algorithm iteration and multi-scenario requirements. 2. Full permutation network (Prior Art 2): Based on Benes network and other non-blocking exchange architectures, it supports arbitrary input-output mapping, but its hardware resource consumption is huge. For example, a 16x16 matrix exchange requires a 256x256 scale Benes network, consuming 1920 2x2 switch units, resulting in significant increase in chip area and power consumption (see CN109445752B).

[0003] Although Benes network achieves non-blocking full permutation through multi-level interconnection and cross-connection, its high number of switch units limits its application in resource-sensitive scenarios. Therefore, existing technologies cannot balance dynamic reconfigurability, low hardware overhead and high energy efficiency, resulting in performance bottlenecks for artificial intelligence processors in complex models and diversified scenarios. The industry urgently needs an exchange solution that balances dynamic reconfigurability and hardware efficiency to support flexible data arrangement while reducing resource overhead, thereby improving the computing efficiency and practicality of artificial intelligence processors. SUMMARY

[0004] The present application aims to provide a tensor data exchange circuit, a data flow processing device and method to support multiple modes of filtering, copying, exchanging, rearranging and transposing of input data.

[0005] To achieve the above purpose, an embodiment of the present application provides a tensor data exchange circuit for a data flow processor, which can include a first non-blocking full permutation component and a second non-blocking full permutation component; the first non-blocking full permutation component and the second non-blocking full permutation component perform block-in data exchange and block-to-block data exchange data format conversion on multiple sub-data blocks split from input data to output data that meets the format requirements of a target component; wherein the first non-blocking full permutation component performs block-in data exchange on the sub-data blocks; the second non-blocking full permutation component performs block-to-block data exchange on multiple sub-data blocks.

[0006] In one embodiment, the data format conversion performed by the switching circuit through the first non-blocking permutation component and the second non-blocking permutation component includes at least one of filtering, copying, swapping, rearranging, and transposing.

[0007] In one embodiment, the first non-blocking permutation component includes a plurality of first non-blocking permutation networks corresponding to the sub-data blocks one-to-one.

[0008] In one embodiment, the first non-blocking permutation component performs the same or different intra-block data swapping on part or all of the input sub-data blocks through the first non-blocking permutation networks.

[0009] In one embodiment, the second non-blocking permutation component includes a second non-blocking permutation network, and the number of input ports of the second non-blocking permutation network is greater than or equal to the number of the first non-blocking permutation networks.

[0010] In one embodiment, the bit width of a single input port of the second non-blocking permutation network is greater than or equal to the total output bit width of a single first non-blocking permutation network.

[0011] In one embodiment, the second non-blocking permutation component includes a plurality of second non-blocking permutation networks, and the second non-blocking permutation networks form a cascaded structure, the first layer of the second non-blocking permutation networks is connected to the first non-blocking permutation component, and the number of input ports of the next layer of the second non-blocking permutation networks is greater than or equal to the number of the previous layer of the second non-blocking permutation networks.

[0012] In one embodiment, in the cascaded structure, the bit width of a single input port of the next layer of the second non-blocking permutation networks is greater than or equal to the total output bit width of a single second non-blocking permutation network of the previous layer.

[0013] In one embodiment, the number of layers of the cascaded structure is greater than or equal to the number of times of splitting of the input data.

[0014] In one embodiment, the second non-blocking permutation component performs the same or different inter-block data swapping on part or all of the input sub-data blocks through each layer of the second non-blocking permutation networks.

[0015] An embodiment of the present application further provides a data stream processing device comprising a tensor data exchange circuit, the device further comprising a control unit; the control unit adjusts the transmission path from input to output of the non-blocking full permutation network in the first non-blocking full permutation component and the second non-blocking full permutation component according to the data format conversion requirement of the component connected with the tensor data exchange circuit through data exchange control instruction, so that the tensor data exchange circuit completes data format conversion through intra-block data exchange and inter-block data exchange.

[0016] In an embodiment, the control unit further comprises generating corresponding control instructions according to the data format conversion requirement and the data source component of the data format to be converted, and controlling the data source component to split the data of the data format to be converted into a plurality of sub-data blocks and then input into the tensor data exchange circuit through the control instructions.

[0017] In an embodiment, the data source component comprises a combination of any two or more of a communication engine, a storage unit, a vector and scalar operation unit, and a matrix operation unit; the tensor data exchange circuit performs data format conversion on the data transmitted between any two of the communication engine, the storage unit, the vector and scalar operation unit, and the matrix operation unit according to the data exchange control instruction.

[0018] An embodiment of the present application further provides a data stream processing method, the method comprising: generating corresponding control instructions according to the data format conversion requirement of the component connected with the tensor data exchange circuit through data exchange control instruction, and according to the data source component of the data format to be converted; adjusting the transmission path from input to output of the non-blocking full permutation network in the first non-blocking full permutation component and the second non-blocking full permutation component in the tensor data exchange circuit through the data exchange control instruction; controlling the data source component to split the data of the data format to be converted into a plurality of sub-data blocks and then input into the tensor data exchange circuit through the control instructions, so that the tensor data exchange circuit completes corresponding data format conversion through intra-block data exchange and inter-block data exchange.

[0019] In an embodiment, generating corresponding control instructions according to the data format conversion requirement and the data source component of the data format to be converted comprises: determining a data splitting strategy according to the storage type of the data of the data format to be converted in the data source component and the data format conversion requirement; and generating corresponding control instructions according to the data splitting strategy.

[0020] In an embodiment, the data splitting strategy comprises splitting according to specified axis granularity and splitting according to block granularity; wherein the splitting according to specified axis granularity comprises splitting according to row granularity direction, splitting according to column granularity direction, and splitting according to channel direction granularity of three-dimensional data.

[0021] An embodiment of the present application further provides a chip comprising the data flow processing device.

[0022] An embodiment of the present application further provides a board card comprising the chip.

[0023] An embodiment of the present application further provides an electronic device comprising the board card or the chip.

[0024] The tensor data exchange circuit, the data flow processing device and the method provided by the present application can significantly improve the dynamic arrangement capability and resource efficiency of an artificial intelligence processor through an innovative block strategy and a reconfigurable network design, and the specific technical advantages include hardware resource efficiency, dynamic multi-mode compatibility, high throughput and low power consumption collaborative optimization, and flexible scalable architecture. Through the exchange mechanism of the "block-global" cooperation, the inherent contradiction between the functional flexibility and the hardware efficiency of the traditional scheme is broken, and the core data arrangement capability of the artificial intelligence processor with high energy efficiency, low delay and strong adaptability is provided for the data flow architecture, which helps the efficient deployment and real-time inference of complex models.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory, but not limiting the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0026] The accompanying drawings described herein are used to provide further understanding of the present application, form a part of the present application, and do not constitute a limitation of the present application. In the drawings:

[0027] Figure 1A A schematic diagram of the principle structure of the tensor data exchange circuit provided by an embodiment of the present application;

[0028] Figure 1B A schematic diagram of the benes network provided by an embodiment of the present application;

[0029] Figure 2 A schematic diagram of the application structure of the tensor data exchange circuit provided by an embodiment of the present application;

[0030] Figure 3 A schematic diagram of the data splitting and conversion process under the cascade structure provided by an embodiment of the present application;

[0031] Figure 4 A schematic diagram of the structure of the data flow processing device provided by an embodiment of the present application;

[0032] Figure 5 A schematic diagram of the flow of the data flow processing method provided by an embodiment of the present application;

[0033] Figure 6 A control instruction generation flowchart provided by an embodiment of the present application;

[0034] Figure 7A and Figure 7B A matrix data splitting logic diagram at different granularities provided by an embodiment of the present application;

[0035] Figure 8A and Figure 8B A matrix data conversion logic diagram under row-column-granularity splitting logic provided by an embodiment of the present application;

[0036] Figures 9A to 9C A matrix data conversion logic diagram under block-granularity splitting logic provided by an embodiment of the present application;

[0037] Figure 10 A matrix data transposition logic diagram under block-granularity splitting logic provided by an embodiment of the present application;

[0038] Figure 11 A structure diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings denote the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0040] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0041] The term "and / or" used herein is merely used to describe associated objects, and indicates that there can be three types of relationships, for example, A and / or B can indicate that there are three cases of A alone, A and B, and B alone. In addition, the term "at least one of" used herein indicates any one of a plurality of or any combination of at least two of a plurality of, for example, includes at least one of A, B, and C, which can indicate any one or more elements selected from the set consisting of A, B, and C.

[0042] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following detailed description. Those skilled in the art will understand that the present disclosure can be implemented without certain specific details. In some examples, methods, means, elements and circuits well known to those skilled in the art are not described in detail in order to highlight the main idea of the present disclosure.

[0043] Please refer toFigure 1A As shown in the embodiment of the present application, a tensor data exchange circuit is provided for a data stream processor, and the exchange circuit comprises a first non-blocking full permutation component and a second non-blocking full permutation component;

[0044] The first non-blocking full permutation component and the second non-blocking full permutation component perform data format conversion on a plurality of sub-data blocks split from input data through intra-block data exchange and inter-block data exchange, to output data meeting the format requirement of a target component;

[0045] The first non-blocking full permutation component performs intra-block data exchange on the sub-data blocks, the second non-blocking full permutation component performs inter-block data exchange on the plurality of sub-data blocks, and the data format conversion comprises at least one of filtering, copying, exchanging, rearranging and transposing.

[0046] In the above embodiment, the first non-blocking full permutation component can comprise a plurality of first non-blocking full permutation networks corresponding to the sub-data blocks one by one, and the first non-blocking full permutation component can comprise at least one second non-blocking full permutation network. It should be noted that the first non-blocking full permutation network and the second non-blocking full permutation network are the same non-blocking full permutation network structure. Different names are used for the convenience of understanding, but they do not represent different structures. Please refer to Figure 1B As shown in the embodiment of the present application, the non-blocking full permutation network can be a Benes network, a Clos network, a strict non-blocking network and a generalized non-blocking network. The connection relationship between the input port and the output port of the non-blocking full permutation network is not unique. Therefore, the intermediate stage cross can be adjusted according to the network structure to change the transmission path of the input and output, so as to realize the input A Figure 1B As shown in the embodiment of the present application, the non-blocking full permutation network can be a Benes network, a Clos network, a strict non-blocking network and a generalized non-blocking network. The connection relationship between the input port and the output port of the non-blocking full permutation network is not unique. Therefore, the intermediate stage cross can be adjusted according to the network structure to change the transmission path of the input and output, so as to realize the input A 0，0 to A 3，3The target component refers to a subsequent processing or storage unit connected to the tensor data exchange circuit and having a specific data format requirement. The target component can include, but is not limited to, a communication engine, a storage unit, a vector and scalar operation unit, a matrix operation unit, etc. For example, in a data flow processing device, a matrix operation unit usually requires input data to be in a two-dimensional matrix format arranged by rows or blocks; while a communication engine may require data to be arranged by channel or a specific interleaving format when sending data; and a storage unit may require data to be arranged in a block interleaving manner when writing data to maximize bandwidth utilization. The tensor data exchange circuit, through the cooperative work of the first non-blocking full permutation component and the second non-blocking full permutation component, splits the input data into multiple sub-data blocks, respectively performs data exchange within and between the blocks, and finally outputs data conforming to the format requirements of the target component, thereby realizing efficient and seamless flow of data between different processing units. The format conversion logic based on the target component will be described in detail in subsequent embodiments, which will not be described one by one here.

[0047] In order to reduce the use of switch units, the application adopts a two-stage non-blocking full permutation component combined with block strategy to realize data format conversion; specifically, taking an actual scenario as an example, for m x m tensor data, a total of m 2 data, if only a non-blocking full permutation network (such as a benes network) is used to exchange the m 2 data, then the input node and the output node need to be m 2 network structure, according to the calculation formula of the non-blocking full permutation network, the number of required 2 x 2 switch units is When m is large, the number of required switch units will be very large; taking m = 16 as an example, the total number of switch units of the non-blocking full permutation network is 1920, if this way is used for circuit implementation, a large amount of logic resources will be used. In order to ensure the correct implementation of the matrix data exchange function and reduce the use of switch units as much as possible, the application splits the original m x m tensor data into m groups of sub-data blocks, then inputs these sub-data blocks into m groups of benes networks respectively, completes the intra-block data exchange, and outputs m groups of vector data with length m; the output vector data with length m is regarded as a whole, and then a benes network with m input ports is used to realize inter-block data exchange, and finally m x m data after exchange is output. From the performance analysis, for m x m tensor data, the consumption of 2 x 2 switch units using this scheme is ; taking m = 16 as an example, the total number of consumed switch units is 952; compared with the existing method, the consumption of 2 x 2 switch units can be reduced by half.

[0048] Specifically, in the above embodiment, to achieve efficient inter-block data exchange, the application adopts a key data packaging and transmission mechanism: the entire length-m vector output by each first non-blocking permutation network is regarded as a complete data block. At the hardware level, the physical bit width of a single input port of the second non-blocking permutation network responsible for inter-block exchange is configured to be greater than or equal to the total bit width of such a data block (i.e., the sum of the bit widths of the m data elements). This enables each data block to be sent as an atomic unit to the second non-blocking permutation network through an input port of the network in one transmission cycle. Please refer to Figure 2 In the above embodiment, the intra-block data exchange and inter-block data exchange mainly rely on the feature that the connection relationship in the non-blocking permutation network can be changed and adjusted. For intra-block data exchange, each sub-data block is input as a one-dimensional vector with a data format of m x m to a first non-blocking permutation network with m input ports according to the order of the elements in the one-dimensional vector. Then, by changing the internal data transmission path of the first non-blocking permutation network, the mapping relationship between the input ports and the output ports can be reconstructed, thereby completing data conversion to generate a new one-dimensional vector and realizing the change of the intra-block element relationship.

[0049] For inter-block data exchange, multiple sub-data blocks correspond to multiple one-dimensional vectors. Each one-dimensional vector corresponding to a sub-data block is input as a whole element to an input port of a second non-blocking permutation network. Thus, n sub-data blocks require the second non-blocking permutation network to have n input ports, so that the relationship between the n sub-data blocks can be changed by changing the internal data transmission path of the second non-blocking permutation network. It should be noted that the intra-block data exchange and the inter-block data exchange do not have a sequence of execution. When the data flow is first through the first non-blocking permutation component and then through the second non-blocking permutation component, the intra-block data exchange is performed first and then the inter-block data exchange is performed. Conversely, when the data flow is first through the second non-blocking permutation component and then through the first non-blocking permutation component, the inter-block data exchange is performed first and then the intra-block data exchange is performed. Thus, more data format conversion requirements can be met. The intra-block data exchange and the inter-block data exchange will be described in detail below with examples of actual data format conversion, which will not be described one by one here.

[0050] In an embodiment of the application, the first non-blocking permutation component performs the same or different intra-block data exchange on part or all of the input sub-data blocks through the first non-blocking permutation network.

[0051] Specifically, in the first non-blocking full permutation assembly, a plurality of the first non-blocking full permutation networks are not required to be used for processing sub-data blocks. For example, there are m first non-blocking full permutation networks in the first non-blocking full permutation assembly, but when there are only m-1 sub-data blocks, only m-1 first non-blocking full permutation networks are used to perform corresponding intra-block data exchange. The number of the first non-blocking full permutation networks is required to be greater than or equal to the number of the split sub-data blocks. In addition, each first non-blocking full permutation network is an independent subject, and there is no consistency requirement between the first non-blocking full permutation networks. In the execution of the above intra-block data exchange process, any two first non-blocking full permutation networks can execute the same or different intra-block data exchange logic. For example, the first non-blocking full permutation network numbered A executes X type of intra-block data exchange, and the first non-blocking full permutation network numbered B executes Y type of intra-block data exchange, where X and Y belong to different exchange logics, that is, the transmission paths from the input port to the output port in the two are not the same. In this way, more fine-grained data exchange control is realized, which is not only more flexible, but also more suitable for different data format conversion scenarios.

[0052] In an embodiment of the present application, the second non-blocking full permutation assembly includes a second non-blocking full permutation network, and the number of input ports of the second non-blocking full permutation network is greater than or equal to the number of the first non-blocking full permutation networks. Further, the bit width of a single input port of the second non-blocking full permutation network is greater than or equal to the total output bit width of a single first non-blocking full permutation network, so as to transmit all output data of any first non-blocking full permutation network as a complete data block.

[0053] Specifically, since each input port of the second non-blocking full permutation network corresponds to a first non-blocking full permutation network, the intra-block data exchange of the sub-data block input by the non-blocking full permutation network is performed. Therefore, the number of input ports of the second non-blocking full permutation network needs to be greater than or equal to the total number of the first non-blocking full permutation networks, so that all the output elements of the first non-blocking full permutation networks can be input to the second non-blocking full permutation network.

[0054] In an embodiment of the present application, the second non-blocking permutation component comprises a plurality of second non-blocking permutation networks, the second non-blocking permutation networks form a cascade structure, the first layer of the second non-blocking permutation networks is connected with the first non-blocking permutation component, and the number of input ports of the second non-blocking permutation networks of the next layer is greater than or equal to the number of the second non-blocking permutation networks of the previous layer. Further, in the cascade structure, the bit width of a single input port of the second non-blocking permutation networks of the next layer is greater than or equal to the total output bit width of a single second non-blocking permutation network of the previous layer, so as to transmit all output data of any second non-blocking permutation network of the previous layer as a complete data block.

[0055] Specifically, referring to FIG. 1, Figure 3 Specifically, referring to FIG. 1, In the embodiment, the second non-blocking permutation network can be determined according to the splitting times of the input data; for example, A data still needs to be further split to complete the fine processing after being split into A1, A2 and A3 once, and can be further split into A11, A12, A13, A21, A22, A23, A31, A32 and A33; at this time, A11, A12, A13, A21, A22, A23, A31, A32 and A33 correspond to a first non-blocking permutation network respectively, and the three first non-blocking permutation networks responsible for converting A11, A12 and A13 perform data format conversion on the first second non-blocking permutation network of the first layer to generate B1, the three first non-blocking permutation networks responsible for converting A21, A22 and A23 perform data format conversion on the second second non-blocking permutation network of the first layer to generate B2, and the three first non-blocking permutation networks responsible for converting A31, A32 and A33 perform data format conversion on the third second non-blocking permutation network of the first layer to generate B3, and then the second non-blocking permutation networks constituting the cascade structure of the first layer are connected with the second non-blocking permutation networks of the cascade structure of the second layer, and the second non-blocking permutation networks of the cascade structure of the second layer perform block data exchange to generate B, so as to realize the process of generating data B from data A after data format conversion; it should be noted that in the above embodiment, the number of layers of the cascade structure is greater than or equal to the splitting times of the input data; for example, if the input data is split twice, the cascade structure has two layers, and similarly, if the input data is split N times, the cascade structure has N layers; a person skilled in the art can further set N plus m layers, m being an integer greater than or equal to 1; in this way, through redundant design, the cascade structure can be applied to more data format conversion scenarios, and the problem that the hardware cannot be adapted due to too many splitting times of the input data can be avoided.

[0056] In actual work, the above cascade structure mainly copes with the multi-stage splitting requirement of input data. For example, for a 100x100 matrix, it can be split into 16 25x25 matrices, and each 25x25 matrix can be further split into 25 5x5 matrices. Accordingly, the cascade structure needs to include at least two layers. Then, the 25 first non-blocking permutation networks are used to perform data exchange on the 25 5x5 matrix expanded sub-data blocks, and the output is output to the second non-blocking permutation network of the first layer for inter-block data exchange. The output data of the last 16 second non-blocking permutation networks of the first layer are output to the second non-blocking permutation network of the second layer, and the inter-block data exchange is performed by the second non-blocking permutation network of the second layer, thereby completing the data format conversion of the 100x100 matrix.

[0057] In the above embodiment, the input port of the second non-blocking permutation network of the subsequent layer is greater than or equal to the number of the second non-blocking permutation network of the previous layer, and the principle is the same as the logic that the input port of the second non-blocking permutation network is greater than or equal to the number of the first non-blocking permutation network. Here, it will not be described one by one.

[0058] In an embodiment of the present application, the second non-blocking permutation assembly performs the same or different inter-block data exchange on part or all of the input sub-data blocks through the second non-blocking permutation network of each layer.

[0059] Specifically, in the second non-blocking permutation assembly, the plurality of second non-blocking permutation networks do not need to be used for processing sub-data blocks. For example, there are m second non-blocking permutation networks in the second non-blocking permutation assembly, but there are only m-1 sub-data blocks, then only m-1 second non-blocking permutation networks are used to perform corresponding inter-block data exchange. Overall, the number of second non-blocking permutation networks needs to be greater than or equal to the number of split sub-data blocks. Secondly, each second non-blocking permutation network is an independent subject, and there is no consistency requirement between the second non-blocking permutation networks. Any two second non-blocking permutation networks can execute the same or different inter-block data exchange logic in the execution of the above inter-block data exchange process. For example, the second non-blocking permutation network numbered A executes X type of inter-block data exchange, and the second non-blocking permutation network numbered B executes Y type of inter-block data exchange, where X and Y belong to different exchange logics, that is, the transmission path from the input port to the output port in the two is not the same. Thus, more fine-grained data exchange control is realized, which is not only more flexible, but also more suitable for different data format conversion scenarios.

[0060] In summary, the tensor data exchange circuit provided in the application significantly improves the dynamic arrangement capability and resource efficiency of an artificial intelligence processor through an innovative blocking strategy and a reconfigurable network design. The specific technical advantages include high hardware resource efficiency, dynamic multi-mode compatibility, high throughput and low power consumption, and a flexible and scalable architecture.

[0061] In an embodiment of the application, a data stream processing device comprising the tensor data exchange circuit is also provided. The device further comprises a control unit. The control unit adjusts the transmission path from input to output of the non-blocking full permutation network in the first non-blocking full permutation component and the second non-blocking full permutation component according to the data format conversion requirement of the components connected to the tensor data exchange circuit, so that the tensor data exchange circuit completes data format conversion through intra-block data exchange and inter-block data exchange.

[0062] In the above embodiment, when the data format conversion requirement is to convert tensor data into matrix data, the transmission path from input to output of the non-blocking full permutation network in the first non-blocking full permutation component and the second non-blocking full permutation component can be adjusted to control the tensor data exchange circuit to perform filtering / copying, transposition and other functions to complete the conversion of tensor data into matrix data or the conversion of matrix data into tensor data. In actual work, the data source component can include a combination of any two or more of a communication engine, a storage unit, a vector and scalar operation unit, and a matrix operation unit. The tensor data exchange circuit performs data format conversion on the data transmitted between any two of the communication engine, the storage unit, the vector and scalar operation unit, and the matrix operation unit according to the data exchange control instruction.

[0063] Specifically, reference can be made to Figure 4 In actual work, the tensor data exchange circuit needs to convert the tensor data received by the communication engine from other processor nodes into the data format required by the matrix operation unit, or convert the matrix data output by the matrix operation unit into tensor data to facilitate the communication engine to send to other processor nodes. For this data stream, the control unit controls the tensor data exchange circuit to complete filtering / copying, transposition and other functions through control instructions.

[0064] In another embodiment, the tensor data exchange circuit needs to convert the matrix data output by the matrix operation unit into vector / scalar data to facilitate the vector / scalar processing unit to process, or reshape the output result of the vector / scalar operation unit into a matrix data format to facilitate subsequent processing of the matrix operation unit. For this data stream, the control unit controls the tensor data exchange circuit to complete filtering / copying, rearrangement and other functions through control instructions.

[0065] In another embodiment, the tensor data exchange circuit needs to convert the tensor data read out from the storage unit in parallel into the data format required by the vector / scalar operation unit; or convert the matrix data output by the vector / scalar operation unit into tensor data in a specific interleaving format, so as to facilitate parallel writing into different blocks of the storage unit with maximum bandwidth. For this data flow, the control unit controls the tensor data exchange circuit to realize filtering / copying, swapping, etc. through control instructions.

[0066] In another embodiment, the tensor data exchange circuit needs to convert the tensor data received by the communication engine from other processor nodes into the data format required by the vector / scalar operation unit; or convert the data output by the vector / scalar operation unit into tensor data, so as to facilitate the communication engine to send to other processor nodes. For this data flow, the control unit controls the tensor data exchange circuit to complete swapping, rearrangement, etc. through control instructions.

[0067] In another embodiment, the tensor data exchange circuit needs to convert the tensor data read out from the storage unit in parallel into the data format required by the matrix operation unit; or convert the matrix data output by the matrix operation unit into tensor data in a specific interleaving format, so as to facilitate parallel writing into different blocks of the storage unit with maximum bandwidth. For this data flow, the control unit controls the tensor data exchange circuit to realize swapping, rearrangement, transposition, etc. through control instructions.

[0068] In another embodiment, the tensor data exchange circuit needs to convert the tensor data received by the communication engine from other processor nodes into the data format required by the vector / scalar operation unit; or convert the data output by the vector / scalar operation unit into tensor data, so as to facilitate the communication engine to send to other processor nodes. For this data flow, the control unit controls the tensor data exchange circuit to complete filtering / copying, swapping, etc. through control instructions.

[0069] It should be noted that the present application needs to split the input data before data format conversion, that is, the control unit further includes a data source component for generating corresponding control instructions according to data format conversion requirements and data formats of the data source component, and inputting the data in the format to be converted into the tensor data exchange circuit after splitting into multiple sub-data blocks by the data source component. The splitting process is mainly realized by the control unit controlling the data source component, and the specific splitting logic in the splitting process will be described in detail in subsequent embodiments, which will not be described one by one here.

[0070] Therefore, the application can support multiple mode input tensor data to output tensor data mapping through the above data flow processing device, significantly improving the dynamic arrangement capability and resource efficiency of the artificial intelligence processor; not only in hardware resource efficiency and dynamic multi-mode compatibility, but also in high throughput and low power consumption optimization and flexible expandable architecture.

[0071] Please refer to Figure 5 The application also provides a data flow processing method in an embodiment of the application, the method comprising:

[0072] S501 According to the data format conversion requirement of the component connected with the tensor data exchange circuit, generate a corresponding control instruction through the data format conversion requirement and the data source component of the data format to be converted;

[0073] S502 Adjust the input to output transmission path of the non-blocking full permutation network in the first non-blocking full permutation component and the second non-blocking full permutation component in the tensor data exchange circuit through the data exchange control instruction;

[0074] S503 Control the data source component to split the data of the data format to be converted into multiple sub-data blocks and input to the tensor data exchange circuit through the control instruction, so as to complete the corresponding data format conversion through the tensor data exchange circuit by using intra-block data exchange and inter-block data exchange;

[0075] Among them, the data format conversion includes at least one of filtering, copying, exchanging, rearranging and transposing.

[0076] In actual work, the execution subject of the above data flow processing method is the control unit mentioned above, and the control instruction in the method can be obtained through the following process, please refer to Figure 6 As shown in the figure, the corresponding control instruction is generated through the data format conversion requirement and the data source component of the data format to be converted, which comprises:

[0077] S601 According to the storage type of the data of the data format to be converted in the data source component and the data format conversion requirement, determine the data splitting strategy;

[0078] S602 Generate a corresponding control instruction according to the data splitting strategy.

[0079] The data splitting strategy includes splitting by specified axis granularity and splitting by block granularity; wherein the splitting by specified axis granularity includes splitting by row granularity, splitting by column granularity and splitting by channel direction granularity of three-dimensional data.

[0080] In this embodiment, the splitting by channel direction granularity of three-dimensional data can be as follows:

[0081] For a 3D tensor data of m x m x m, it can be unfolded into 1D vectors in two ways. Way 1: split into m x m 2D tensors according to the granularity of row (X), column (Y) or the third axis (Z), and then split into 1D vectors according to the 2D tensor splitting method. Way 2: split into sub-3D tensor data according to the granularity of block, for example, the 3D tensor data of m x m x m can be split into m sub-tensor blocks of m x m x m, and each sub-tensor block is unfolded into m groups of 1D vectors of length m according to the row, column or third axis priority. Similarly, higher-dimensional tensor data can also use similar schemes, which will not be described one by one here.

[0082] For the convenience of more clearly understanding the data splitting logic provided by the present application, the above data splitting strategy is schematically described by taking a 2D tensor as an example. It can be known by those skilled in the art that the splitting strategy is only for the convenience of understanding the above data splitting logic, and does not limit it in any way.

[0083] For a matrix data (2D tensor) of m x m, it can be unfolded into 1D vectors in two ways. Way 1: split into 1D vectors according to the granularity of row or column; for a matrix of m x m, it can be split into m groups of 1D vectors of length m. Way 2: split into 1D vectors according to the granularity of block, split the matrix data of m x m into m sub-matrix blocks of m x m, and each sub-matrix block is unfolded into m groups of 1D vectors of length m according to the row priority or column priority. For example, Figure 7A and Figure 7B show two ways of unfolding the original matrix data of 4 x 4, in which the row / column granularity ( Figure 7A ) is to unfold the original matrix data into 4 groups of vectors according to the row or column; and the block granularity ( Figure 7B ) is to unfold the original matrix data into 4 groups of vectors according to the 2 x 2 sub-block.

[0084] By controlling the way of unfolding the matrix data into vectors, the filtering, copying, swapping, rearranging and transposing functions of the tensor data can be realized; in the actual use process, the dynamic configuration of the unfolding way can be realized through software programming. When way 1 is adopted, the data filtering, copying and swapping are based on the original matrix data according to the row or column, and the rearrangement of the original matrix data can also be realized according to the row or column as the granularity. When way 2 is adopted, the data filtering, copying and swapping are based on the sub-data block in the original matrix data, and the rearrangement of the original matrix data can also be realized according to the sub-data block as the granularity, and the transposition of the original matrix data can also be realized.

[0085] Take m x m matrix (two-dimensional tensor) data as an example, the filtering, copying, swapping, rearranging and transposing processing principles of tensor data are explained; the subscripts i and j represent the row number and column number of the matrix, and the letter A represents the matrix element, so a specific element in the matrix can be represented as A i，j ;

[0086] The filtering operation means that some elements in the tensor data are set to 0 according to a certain pattern, that is, the tensor is sparsified.

[0087] Suppose the filtering pattern is: :

[0088] The filtering operation can only be implemented in block granularity, and the filtering operation is as follows:

[0089] ;

[0090] Therefore, the corresponding filtering operation is completed on the two-dimensional tensor data by the filtering pattern, and in this process, A i0,j0 represents the first data in the first row and the first column, and A in,j0 is the data in the nth row and the first column, and A in,jn is the data in the nth row and the nth column. The copying operation means copying some elements, rows or columns in the tensor data;

[0091] Single element copying:

[0092] ;

[0093] Copy by row, copy some row data into a two-dimensional tensor:

[0094] ;

[0095] Copy by column, copy some column data into a two-dimensional tensor:

[0096] ;

[0097] Support data swapping operation of any tensor element within a block:

[0098] ;

[0099] Tensor data rearrangement operation, such as row rearrangement or column rearrangement:

[0100] ;

[0101] ;

[0102] Tensor data transposition operation, the matrix shape becomes m x m after transposition;

[0103] ;

[0104] Based on the above filtering, copying, swapping, rearranging and transposing processing principles, taking a 4x4 size matrix data as an example, please refer to the figure as shown in Figure 8A , the flow of implementing matrix data filtering and copying by using mode 1 is as follows:

[0105] When using data filtering processing: by filtering the first row, the elements A 0，1 , A 0，2 and A 0，3 in the first row of the matrix are filtered to 0, and other elements remain original values.

[0106] When performing element copying: the element A 0，1 in the first row of the matrix is copied to other positions of the row.

[0107] When performing row copying: the elements of the 0th row are copied and filled into the 1st, 2nd and 3rd rows.

[0108] When performing column copying: the elements of the 3rd column are copied and filled into the 0th, 1st and 2nd columns.

[0109] Please refer to the figure as shown in Figure 8B , the flow of swapping and rearranging functions is as follows:

[0110] When performing data swapping: the element A 0，1 in the 0th row and 1st column is swapped with the element A 0，3 in the 0th row and 3rd column.

[0111] When performing row rearrangement: all elements of the 0th row are swapped with all elements of the 2nd row.

[0112] When performing column rearrangement: all elements of the 1st column are swapped with all elements of the 3rd column.

[0113] Thus, the matrix data filtering, copying, swapping and rearrangement functions can be implemented according to the modes shown in Figure 8A and Figure 8B ; in actual work, these functions can be used in combination or individually, and the present application does not make any limitation here.

[0114] There are some differences between mode 1 and mode 2 when implementing matrix data filtering, copying, swapping, rearrangement and transposition functions, which can be referred to the figure as shown in Figure 9A , the flow of implementing matrix data filtering and copying by using mode 2 is as follows:

[0115] When performing data filtering: the elements A 1，0 and A 0，1 in the sub-block 0 of the matrix are filtered to 0, and other elements remain original values.

[0116] When performing element copy: copy the element A 0，1 in sub-block 0 in the matrix to other positions of the sub-block;

[0117] When performing block copy: copy and fill the element of sub-block 0 into sub-blocks 1, 2, and 3.

[0118] Referring to FIG. 6, the flow of data exchange and block rearrangement is as follows: Figure 9B

[0119] Data exchange: exchange the element A 0，1 in sub-block 0 with the element A 1，0 ;

[0120] Block rearrangement: exchange all elements of sub-block 0 with all elements of sub-block 3 in corresponding positions.

[0121] Referring to FIG. 7, the flow of transposition operation is as follows: Figure 9C

[0122] Transposition operation: according to the matrix transposition operation rule, exchange and rearrange all elements in the 4x4 matrix to obtain the result after transposition of the matrix. The specific implementation manner can refer to FIG. 8. Figure 10

[0123] The data stream processing method provided in the present application significantly improves the dynamic arrangement capability and resource efficiency of an artificial intelligence processor through an innovative block strategy and a reconfigurable network design. Through the exchange mechanism of “block-global” cooperation, the inherent contradiction between functional flexibility and hardware efficiency of the traditional scheme is broken through, and the core data arrangement capability with high energy efficiency, low delay, and strong adaptability is provided for the artificial intelligence processor of the data stream architecture, which helps efficient deployment and real-time inference of complex models.

[0124] In an embodiment of the present application, a chip is also provided, which comprises the data stream processing apparatus.

[0125] In an embodiment of the present application, a board card is also provided, which comprises the chip.

[0126] Since the chip and the board card are similar to the data stream processing method in terms of the principle of solving problems, the implementation of the chip and the board card can refer to the implementation of the data stream processing method, and the repeated parts will not be described again.

[0127] ​​​In an embodiment of the present application, an electronic device is also provided, which is deployed with the board card or the chip.

[0128] Figure 11 is a block diagram of an electronic device 900 for implementing the above data processing method according to an exemplary embodiment. For example, the electronic device 900 can be an AI server, a training-computing integrated machine, etc.

[0129] With reference to Figure 11 The electronic device 900 can include one or more of the following components: an AI acceleration computing module, a CPU module, a power supply module, a hard disk module, and a fan module. Each module works cooperatively through a standardized hardware interface and a bus system, and the specific architecture is as follows:

[0130] The AI acceleration computing module includes a plurality of parallel deployed AI acceleration cards, each of which integrates at least one AI acceleration chip (such as an RPU chip, a GPU chip, a CGRA chip, etc.). The AI acceleration cards are connected through a high-speed inter-card direct connection (C2C, Card-to-Card) interconnection structure to realize data communication, supporting low-latency and high-bandwidth horizontal expansion. The AI acceleration chip is dedicated to performing AI computing tasks such as high-density matrix operation, neural network model training and / or inference, etc., providing main computing power support.

[0131] The CPU module includes at least one CPU board card, which carries a central processing unit (CPU) chip and a matching CPU memory (such as DDR4 / DDR5, RAM) on the board card. The CPU chip serves as the system control center, responsible for task scheduling, resource allocation, I / O management, and coordination of parallel operation of the AI acceleration computing module, while handling non-accelerated general computing tasks.

[0132] The power supply module is configured with a redundant power supply unit to provide stable power distribution and management for the AI acceleration computing module, the CPU module, and other modules.

[0133] The hard disk module integrates a high-speed solid state drive (SSD) and / or a large-capacity mechanical hard disk (HDD) and is connected to the system bus through a backplane. The hard disk stores an operating system, AI training data sets, model parameters, and computing results, providing a high-throughput data read-write channel and supporting data preprocessing and persistence.

[0134] The fan module uses a multi-zone independent speed control fan array and is configured in a key heat source area (such as an AI acceleration card, a CPU heat dissipation area), which realizes system heat dissipation through forced air cooling and guarantees stable operation of high-efficiency computing components.

[0135] The CPU module is connected to the AI acceleration computing module through a PCIe bus to realize task issuance, result recovery, and memory coordination.

[0136] The CPU module manages the data access of the hard disk module through the SATA / SAS / NVMe interface.

[0137] The power module provides hierarchical power supply to all functional modules through the power distribution backplane.

[0138] The fan module adjusts the fan speed according to the temperature monitoring signal from the CPU board card and the AI acceleration card.

[0139] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be referred to each other, and for the sake of brevity, they will not be described herein.

[0140] Those skilled in the art can understand that in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined by its function and possible inherent logic.

[0141] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical application or improvement of technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A tensor data exchange circuit for a data stream processor, characterized in that, The switching circuit includes a first non-blocking full-arrangement component and a second non-blocking full-arrangement component; The first non-blocking full permutation component and the second non-blocking full permutation component are used to perform intra-block data exchange and inter-block data exchange on multiple sub-data blocks split from the input data, so as to output data that meets the format requirements of the target component. Wherein, the first non-blocking full permutation component performs intra-block data exchange on the sub-data blocks; the second non-blocking full permutation component performs inter-block data exchange on multiple sub-data blocks; the first non-blocking full permutation component includes multiple first non-blocking full permutation networks corresponding one-to-one with the sub-data blocks; the second non-blocking full permutation component includes a second non-blocking full permutation network.

2. The tensor data exchange circuit according to claim 1, characterized in that, The data format conversion performed by the switching circuit through the first non-blocking permutation component and the second non-blocking permutation component includes at least one of filtering, copying, swapping, rearranging, and transposing.

3. The tensor data exchange circuit according to claim 1, characterized in that, The first non-blocking permutation component performs intra-block data exchange on some or all of the input sub-data blocks through the first non-blocking permutation network, either with the same or different data.

4. The tensor data exchange circuit according to claim 1, characterized in that, The number of input ports of the second non-blocking permutation network is greater than or equal to the number of the first non-blocking permutation networks, and the bit width of a single input port of the second non-blocking permutation network is greater than or equal to the total output bit width of a single first non-blocking permutation network.

5. The tensor data exchange circuit according to claim 1, characterized in that, The second non-blocking full permutation component includes multiple second non-blocking full permutation networks, which form a cascaded structure. The first layer of the second non-blocking full permutation network is connected to the first non-blocking full permutation component, and the number of input ports of the second non-blocking full permutation network in the next layer is greater than or equal to the number of the second non-blocking full permutation networks in the previous layer. In the cascaded structure, the bit width of a single input port of the second non-blocking full permutation network in the next layer is greater than or equal to the total output bit width of a single second non-blocking full permutation network in the previous layer.

6. The tensor data exchange circuit according to claim 5, characterized in that, The number of layers in the cascaded structure is greater than or equal to the number of times the input data is split.

7. The tensor data exchange circuit according to claim 5, characterized in that, The second non-blocking permutation component performs the same or different inter-block data exchange on some or all of the input sub-data blocks through each layer of the second non-blocking permutation network.

8. A data stream processing apparatus comprising a tensor data exchange circuit according to any one of claims 1 to 7, characterized in that, The device also includes a control unit; The control unit adjusts the input-to-output transmission path of the non-blocking permutation network in the first and second non-blocking permutation components according to the data format conversion requirements of the components connected to the tensor data exchange circuit, so that the tensor data exchange circuit can complete the data format conversion through intra-block data exchange and inter-block data exchange.

9. The data stream processing apparatus according to claim 8, characterized in that, The control unit also includes generating corresponding control instructions based on the data format conversion requirements and the data source component of the data format to be converted. The control instructions control the data source component to split the data of the format to be converted into multiple sub-data blocks and then input them into the tensor data exchange circuit.

10. The data stream processing apparatus according to claim 9, characterized in that, The data source component includes any combination of two or more of the following: a communication engine, a storage unit, a vector and scalar operation unit, and a matrix operation unit. The tensor data exchange circuit performs data format conversion on any two of the communication engine, the storage unit, the vector and scalar operation unit, and the matrix operation unit according to the data exchange control command.

11. A data stream processing method, characterized in that, The method includes: Based on the data format conversion requirements of the components connected to the tensor data exchange circuit, the corresponding control instructions are generated by the data format conversion requirements and the data source components of the data format to be converted. The data exchange control command adjusts the input-to-output transmission path of the non-blocking permutation network in the first and second non-blocking permutation components of the tensor data exchange circuit. The control command controls the data source component to split the data to be converted into multiple sub-data blocks and input them into the tensor data exchange circuit, so that the tensor data exchange circuit can complete the corresponding data format conversion by using intra-block data exchange and inter-block data exchange.

12. The data stream processing method according to claim 11, characterized in that, The generation of corresponding control instructions based on the data format conversion requirements and the data source component of the data format to be converted includes: The data splitting strategy is determined based on the storage type of the data to be converted in the data source component and the data format conversion requirements; The corresponding control instructions are generated based on the data splitting strategy.

13. The data stream processing method according to claim 12, characterized in that, The data splitting strategy includes splitting by specified axis granularity and splitting by block granularity; The splitting by specified axis granularity includes splitting by row granularity, splitting by column granularity, and splitting by channel granularity of three-dimensional data.

14. A chip, characterized in that, The chip includes the data stream processing apparatus according to any one of claims 8 to 10.

15. A circuit board, characterized in that, Includes the chip as described in claim 14.

16. An electronic device, characterized in that, The board as described in claim 15 or the chip as described in claim 14 is deployed.

Images