Packet network capable of sequential packet transmission without loss of packets

Abstract

The present specification discloses an arithmetic processing device that realizes high throughput at low cost. The arithmetic processing device includes an instruction fetch unit that reads data required for an operation for executing processing of a neural network from a memory and provides the data to an operation unit, the instruction fetch unit including: a plurality of routers having a data processing mapping table that describes a manner of processing input data according to a node ID of the input data; and an instruction fetch network controller that controls each data processing mapping table that the plurality of routers have, each router including an arbitration section that controls an input port according to tail information of input data and changes the tail information according to tail maintenance information of the input data.

Description

Technical Field

[0001] This invention relates to neural network processors, and more particularly to networks for transmitting packets. Background Technology

[0002] This application claims priority to Korean Patent Application No. 10-2023-0154008, filed on November 8, 2023, the contents of which, in its specification and drawings, are incorporated herein by reference.

[0003] Artificial Neural Networks (ANNs) are a technology that achieves artificial intelligence by interconnecting artificial neurons that are mathematically modeled after neurons in the human brain. A mathematical model of an artificial neuron is shown in formula (1) below. Specifically, an artificial neuron receives an input signal x. i and x i Each with its corresponding weight w i After multiplying, the results are summed. Next, the artificial neuron uses an activation function to calculate the activation value and passes it to the next connected artificial neuron.

[0004] Formula (1)

[0005] y = f(w1 * x1 + w2 * x2 + .... w n * x n ) = f(Σ w i * x i ), where i = 1......n, n = input signal number.

[0006] As a form of ANN, Deep Neural Networks (DNNs) have a layered network architecture where artificial neurons (nodes) are arranged in layers. A DNN consists of an input layer, an output layer, and multiple hidden layers between the input and output layers. The input layer consists of multiple nodes that receive input values. These nodes pass the output value, calculated using the mathematical model described above, to the next hidden layer node connected to the input layer. Hidden layer nodes receive input values ​​using the mathematical model, calculate the output value, and then pass that output value to the output layer node.

[0007] As a form of machine learning implemented in DNNs, the operation of deep learning can be divided into two processes: the training process, which involves continuously learning from learning data to improve the computational power of a given DNN; and the inference process, which uses the DNN trained through the training process to infer new input data.

[0008] Deep learning inference employs forward propagation, where input data is received by nodes in the input layer and then processed sequentially in the hidden and output layers. Finally, the nodes in the output layer derive the inference conclusion based on the output values ​​of each hidden layer.

[0009] In contrast, during deep learning training, to reduce the discrepancy between the inference conclusion and the true answer, the weights of nodes are adjusted. Typically, weights are adjusted using gradient descent. To implement gradient descent, the differential value of the difference between the inference conclusion and the true answer needs to be calculated for each node's weight. In this process, the differential value of the weights of the front-end nodes of the DNN is calculated using the chain rule of the differential values ​​of the weights of the back-end nodes. Since the chain rule is calculated in the reverse direction of the inference process, deep learning employs backpropagation.

[0010] In other words, DNN has a hierarchical structure. Nodes in each layer receive input results from multiple nodes in the previous layer, perform operations based on the mathematical model of the aforementioned nodes to output new results, and pass the new results to nodes in the next layer.

[0011] Furthermore, DNNs can employ a distributed processing architecture, which distributes the numerous computations performed by nodes in each layer across multiple processing units. Each processing unit loads the necessary data from memory, performs the computation, and then stores the results back into memory. Summary of the Invention

[0012] The technical problem that the invention aims to solve

[0013] The purpose of this specification is to provide a packet network that can transmit packets in the correct order without losing data packets.

[0014] This specification is not limited to the issues mentioned above, and those skilled in the art will clearly understand from the following description other issues not mentioned in this specification.

[0015] Technical solutions for solving the problem

[0016] The present specification provides an arithmetic processing apparatus for solving the above-mentioned problems, wherein the arithmetic processing apparatus includes an instruction fetching unit, which reads data required for computation from memory and provides it to a computation unit for performing neural network processing. The instruction fetching unit includes: a plurality of routers having a data processing mapping table, the data processing mapping table recording the method of processing the input data according to the node ID of the input data; and an instruction fetching network controller, used to reconstruct the data processing mapping tables of the plurality of routers to form a software topology according to the computation form. Each router includes an arbitration unit, the arbitration unit being used to control the input port according to the tail information of the input data and to change the tail information according to the tail maintenance information of the input data.

[0017] According to one embodiment of this specification, each of the routers may include: a main input port for inputting data from the memory, a first transmission output port for transmitting data to a neighboring first router, a first transmission input port for inputting data transmitted from a neighboring first router, a second transmission output port for transmitting data to a neighboring second router, a second transmission input port for inputting data transmitted from a neighboring second router, and a main output port for providing data to the arithmetic unit.

[0018] According to one embodiment of this specification, the data processing mapping table can be used to store information about whether to perform input data blocking, reflection, and output.

[0019] According to one embodiment of this specification, the instruction fetching network controller can be used to set whether to perform the blocking, reflection, and output based on the topology to be reconstructed.

[0020] According to one embodiment of this specification, the instruction fetching network controller can be used to set the blocking and output in the data processing mapping table of routers belonging to the same group within the reconstructed topology to be the same.

[0021] According to one embodiment of this specification, the memory may include a number of data storage fragments equal to the number of the plurality of routers.

[0022] According to one embodiment of this specification, the instruction fetching unit may include: a network interface for retrieving data stored in the memory; an instruction fetching network for transmitting the data retrieved to the network interface; and a feed module for providing the data transmitted from the instruction fetching network to the computing unit. In this case, the plurality of routers and the instruction fetching network controller may be included in the instruction fetching network.

[0023] According to one embodiment of this specification, the network interface may include: an instruction fetch buffer for storing fetched data; and an interface controller for assigning node IDs corresponding to each data storage shard to the fetched data.

[0024] According to one embodiment of this specification, the tail information may be any one of a first logical value T representing the end of a data packet and a second logical value F representing a non-end of a data packet, and the tail maintenance information may be any one of a first logical value T representing maintaining the tail information and a second logical value F representing changing the tail information.

[0025] According to one embodiment of this specification, the arbitration unit can be used to change the input port to input data from a neighboring router when the control input port is used to input data from the memory, if the tail information of the input data from the memory is a first logic value T.

[0026] According to one embodiment of this specification, the arbitration unit can be used to change the tail information of the input data to the second logic value F when the tail information of the input data from the memory is maintained at the second logic value F.

[0027] According to one embodiment of this specification, the arbitration unit can be used to control the input data from the memory when the tail information of the input data from the neighboring router is a first logic value T.

[0028] According to one embodiment of this specification, the interface controller can be used to further set the tail information and the tail maintenance information on the retrieved data.

[0029] According to one embodiment of this specification, the interface controller can be used to set the tail information of the last data of the extracted data packet to a first logical value T, and set the tail information of the remaining data to a second logical value F.

[0030] According to one embodiment of this specification, the interface controller can be used in a software topology of a router constructed by the instruction fetch network controller based on the data processing mapping table to set the tail maintenance information of data with the last node ID to a first logical value T representing the maintenance of tail information, and set the tail maintenance information of data with the remaining node IDs to a second logical value F representing the change of tail information.

[0031] Other specific aspects of this invention are detailed in the description and accompanying drawings.

[0032] Invention Effects

[0033] According to one aspect of this specification, even if the router experiences changes in buffer backpressure or router delay, it can be ensured that no data packets are lost and that data packets are transmitted in order.

[0034] According to another aspect of this specification, a low-cost, high-throughput computing device can be provided. Furthermore, it can achieve superior computing power compared to conventional neural network processors.

[0035] The effects of the present invention are not limited to those mentioned above, and those skilled in the art can clearly understand other effects not mentioned from the following description. Attached Figure Description

[0036] Figure 1 This is a block diagram that schematically illustrates the structure of a computational processing apparatus according to an embodiment of the present invention.

[0037] Figure 2 This is a block diagram showing in more detail the various structures of the arithmetic processing apparatus according to this specification.

[0038] Figure 3 This is a block diagram showing more specifically the structure of an instruction fetch unit according to one embodiment of this specification.

[0039] Figure 4 This is a reference diagram used to illustrate the structure of the router according to this specification.

[0040] Figure 5 It is the software topology according to the first embodiment.

[0041] Figure 6 This is a reference diagram of the data processing mapping table according to the first embodiment.

[0042] Figure 7 It is the software topology according to the second embodiment.

[0043] Figure 8This is a reference diagram of the data processing mapping table according to the second embodiment.

[0044] Figure 9 It is a software topology based on the third embodiment.

[0045] Figure 10 This is a reference diagram of the data processing mapping table according to the third embodiment.

[0046] Figure 11 It is the software topology according to the fourth embodiment.

[0047] Figure 12 This is a reference diagram of the data processing mapping table according to the fourth embodiment.

[0048] Figure 13 This is an example diagram of the data input timing according to one embodiment of this specification.

[0049] Figures 14 to 20 This is a reference diagram showing the data processed by the arbitration department. Detailed Implementation

[0050] The advantages, features, and methods of implementing the inventions disclosed herein will become clear with reference to the accompanying drawings and the detailed embodiments described below. However, this specification is not limited to the embodiments disclosed below, but can be implemented in many different forms. These embodiments are provided only to make the disclosure of this specification more complete and to fully demonstrate the scope of this specification to those skilled in the art (hereinafter referred to as "those skilled in the art"), the scope of which is defined only by the claims.

[0051] The terminology used in this specification is for illustrative purposes only and is not intended to limit the scope of this specification. In this specification, unless otherwise specified, the singular form includes the plural form. The use of "comprises" and / or "comprising" in this specification does not exclude the presence or addition of more than one other structural element besides the described structural element.

[0052] Throughout this specification, the same reference numerals denote the same structural elements, and "and / or" includes each of the structural elements as well as all combinations thereof. Although "first," "second," etc., are used to describe various structural elements, these structural elements are clearly not limited by these terms. These terms are only used to distinguish one structural element from another, and it is clear that the first structural element mentioned below, within the scope of the technical concept of this invention, can also be considered a second structural element.

[0053] Unless otherwise defined, all terms used in this specification (including technical and scientific terms) shall be interpreted as commonly understood by one of ordinary skill in the art to which this specification pertains. Furthermore, terms already defined in commonly used dictionaries should not be idealized or over-interpreted unless explicitly defined. Embodiments of the invention will now be described in detail with reference to the accompanying drawings.

[0054] During the training process of deep learning, the data used can be in the form of tensors ranging in size from hundreds of kilobytes to hundreds of megabytes. This data can be stored in multiple memory banks that make up on-chip memory.

[0055] Multiple memory banks and multiple computing units are connected via a network for data transmission. In the case of a network-on-chip (NAT) architecture, the network can be built inside the chip and may include routers. A router includes a device capable of passing data packets from multiple nodes to multiple nodes. For data packets (traffic) entering from multiple directions, the router can perform at least one of the following actions: i) passing to the destination direction; ii) arbitrating in the event of contention; and iii) performing flow control to prevent packet loss. The performance and cost of such a router are determined by topology, bandwidth, flow control, buffers, etc., aiming to achieve high throughput at a low cost (cost, area, and energy consumption).

[0056] On the other hand, in deep learning, most traffic patterns involve reusing the same tensor multiple times to generate multiple output tensors. Therefore, to reduce memory accesses, routers can fetch input tensor data from memory and broadcast or multicast it to multiple computational units. A common method for multicast is to record all destinations for each data packet and transmit it. This method has a problem: as the number of nodes increases, the packet header size grows proportionally (e.g., if the packet header includes a bitmap to identify the destination, at least 64 bits are needed for 64 nodes). Typically, buffered flow control suffers from head-of-line blocking due to buffer size. One solution is source throttling, but this method only detects and avoids congestion after it has occurred. Therefore, considering the routing pattern characteristics of deep learning, a network that can achieve high throughput at low cost is needed.

[0057] Figure 1 This is a block diagram that schematically illustrates the structure of a computational processing device according to an embodiment of the present invention.

[0058] like Figure 1 As shown, the arithmetic processing device 10 may include a memory 100, a fetch unit 200, an operation unit 300, and a commit unit 400. However, as... Figure 1 As shown, the arithmetic processing device 10 does not necessarily include all of the memory 100, instruction fetch unit 200, arithmetic unit 300, and submit unit 400. For example, the memory 100 and submit unit 400 can be arranged outside the arithmetic processing device 10.

[0059] The memory 100 is capable of storing at least one of the data described in this specification. For example, the memory 100 can store input data, tensors, output data, filters, operation result data of the arithmetic unit, all data used by the instruction fetch unit, etc. The memory 100 can be configured as a data memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), but it is not mandatory.

[0060] The instruction fetch unit 200 can read the data required for the operation from the input data stored in the memory 100 and provide it to the arithmetic unit 300. When the input data is a tensor, the instruction fetch unit 200 can read the tensor stored in the memory 100 and feed it to the arithmetic unit 300 according to the operation type. The operation type may include, for example, matrix multiplication, convolution, grouped convolution, etc. In this case, the instruction fetch unit 200 can sequentially read data groups from the memory 100 that are equal to or greater than the unit data processing capacity of one or more arithmetic units of the arithmetic unit 300 and supply them to the arithmetic unit 300.

[0061] The arithmetic unit 300 can perform arithmetic processing on the input data received from the instruction fetch unit 200 to generate output data. The arithmetic unit 300 can be configured according to (or corresponding to) the type of operation to be performed. For example, the arithmetic unit 300 can process the data supplied from the instruction fetch unit 200 in a streaming manner, but is not limited thereto. The arithmetic unit 300 may include more than one arithmetic unit.

[0062] The submission unit 400 can store the computation result data output from the computation unit 300 (e.g., in a streaming manner) into the memory 100. When storing the computation result data received from the computation unit 300 into the memory 100, the submission unit 400 can store the computation result data into the memory 100 according to the type of computation to be performed subsequently. For example, the submission unit 400 can convert the computation result data into a preset format or a format suitable for subsequent computations before storing it in the memory 100.

[0063] Figure 2 This is a block diagram showing in more detail the various structures of the arithmetic processing device according to this specification.

[0064] Reference Figure 2 The memory 100, instruction fetch unit 200, arithmetic unit 300 and submit unit 400 described above will be explained in more detail.

[0065] The memory 100 can be configured according to a memory address space. For example, the memory address space can be contiguous or sequential. Furthermore, the memory address space can be one-dimensional, but is not limited to this; it can also be an array of two or more dimensions. The internal structure of the memory 100 can be configured as an independently accessible slice structure. For example, the memory 100 can include multiple data storage slices 101. The number of data storage slices 101 can be determined based on the number of dot product engines 310 included in the arithmetic unit 300. For example, the number of slices 101 can be the same as the number of dot product engines 310 included in the arithmetic unit 300. For example, when the input data is a tensor, the tensor can be segmented along the channel direction and the height direction, and then stored in the data storage slice 101.

[0066] The instruction fetch unit 200 can read data from the memory 100 and provide that data to the dot product engine 310 of the computation unit 300. For example, the instruction fetch unit 200 may include at least one of an instruction fetch sequencer 210, a network interface 220, an instruction fetch network 230, a supply module 240, and a computation sequencer module 250. The instruction fetch sequencer 210 can control the data fetching operation from the memory 100 to the network interface 220. Since the network interface 220 is used to retrieve data stored in the memory 100, it provides an interface between the memory 100 and the instruction fetch network 230. The instruction fetch network 230 can transmit the retrieved data to the supply module 240. The computation sequencer module 250 can control the computation unit 300 to perform a specified operation by controlling the supply module 240 and the data input to the supply module 240.

[0067] The instruction fetch network 230 of the instruction fetch unit 200 can have various structures depending on the operation content and data format. The instruction fetch network 230 can be configured or reconstructed by software into a topology required by the operation unit 300. Furthermore, the instruction fetch network 230 can determine its topology based on the shape of the input data and the type of operation. The instruction fetch network 230 can support various communication modes, such as Direct, Vertical Multicast, Channel Multicast, and Vertical Nearest Neighbor, depending on the operation performed by the operation unit 300, but is not limited to these.

[0068] For example, in the case of two-dimensional convolution, it is assumed that the values ​​of all input channels need to be input to the dot product engine 310 used to calculate the individual output activations. Therefore, the instruction fetching unit 200 can provide the input activation values ​​read sequentially along the channel direction to the dot product engine 310 in a multicast manner. Furthermore, the instruction fetching unit 200 can use the instruction fetching sequencer 210 to sequentially read the data to be input from each data storage segment 101 to the computation unit 300. The data read from the data storage segment 101 by the instruction fetching sequencer 210 can be passed to the computation unit 300 via the instruction fetching network 230 of the instruction fetching unit 200.

[0069] Thus, the instruction fetch unit 200 can fetch tensor slices from the memory 100 in parallel and provide them to the arithmetic unit 300 in a form that the arithmetic unit 300 can perform operations on. Here, the instruction fetch network 230 may further include an instruction fetch network controller (…). Figure 2 (Not shown), which is used to form and manage the instruction fetch network 230 so that data fetched from memory 100 can be transferred to the arithmetic unit 300 that needs the data.

[0070] The computation unit 300 may include multiple parallelizable dot product engines (310). For example, the computation unit 300 may include 256 dot product engines 310, but is not limited thereto. Each dot product engine 310 may include more than one arithmetic unit (e.g., 32 MACs). Each dot product engine 310 can perform multiple operations depending on the configuration of the arithmetic unit. The dot product engines 310 of the computation unit 300 can also be segmented along the channel and height directions and perform operations to generate output activations.

[0071] In addition to the dot product engine 310, the arithmetic unit 300 may further include a register file (not shown).

[0072] The register file is a storage space used by the dot product engine 310 to temporarily store one of the relatively frequently used or reusable operands during operation. For example, the register file can be configured as SRAM or DRAM, but is not limited to these.

[0073] For example, when performing operations in a neural network, for a regular convolutional layer with a large activation size, the weights can be stored in a register file, while the activations can be stored in memory. Furthermore, in the case of a fully connected layer with an activation size larger than the weight size, the weights can be stored in memory, while the activations are stored in a register file.

[0074] For example, when the arithmetic unit 300 performs a MAC operation, the dot product engine 310 can be used as an operand for the MAC operation, utilizing input data from the instruction fetch unit 200, register values ​​from the register file located in the dot product engine 310, and accumulated values ​​from the accumulator. Furthermore, the operation result can be stored back in the accumulator or passed as output data to the commit unit 400 for storage in the memory 100.

[0075] On the other hand, as described above, the submission unit 400 can convert the output activation value calculated by the arithmetic unit 300 into the form required for subsequent operations and store it in the memory 100.

[0076] For example, in a neural network, in order to make the output activation values ​​generated by the operation at a specified layer usable for the operation at the next layer, the submission unit 400 stores the output activation values ​​in memory. In addition, depending on the data format required for the operation at the next layer, the submission unit 400 can perform a transpose operation (e.g., tensor manipulation) and pass the result data to memory 100 for storage through the submission network (not shown).

[0077] Thus, after the computation unit 300 calculates the data, the submission unit 400 stores the output data in the memory 100 in the required form. In order to store the output data in the required form, the submission unit 400 can perform a tensor transpose operation using a data transpose module (not shown) and a submission network module (not shown).

[0078] Figure 3 This is a block diagram showing more specifically the structure of an instruction fetch unit according to one embodiment of this specification.

[0079] Reference Figure 3 It can be confirmed that the instruction fetching unit 200 according to one embodiment of this specification includes a network interface 220, an instruction fetching network 230, and a feed module 240.

[0080] Data stored in each data storage shard 101 can be retrieved to the network interface 220. The network interface 220 may include: an instruction fetch buffer 222, which stores the retrieved data; and an interface controller 221, which assigns the retrieved data a node ID corresponding to each data storage shard.

[0081] The instruction fetch network 230 is capable of transmitting data fetched to the instruction fetch buffer 222. Therefore, the instruction fetch network 230 may include multiple routers 232 and an instruction fetch network controller 231.

[0082] Each of the plurality of routers 232 may have a data processing mapping table. The data processing mapping table can indicate the routing / flow control method (e.g., blocking, reflection, or output) of the input data based on the node ID of the input data. The instruction fetching network controller 231 can reconstruct the data processing mapping table. The data processing mapping table can be adaptively reconstructed according to the type of operation that will be performed on the data subsequently. For example, the instruction fetching network controller 231 can reconstruct the various data processing mapping tables of the plurality of routers 232 into a software-defined topology structure based on the operation pattern. The data processing mapping table will be described in more detail later.

[0083] The feed module 240 can provide data received from the instruction fetching network 230 to the processing unit 300. To this end, the feed module 240 may include a feed buffer 242 that stores data output from a plurality of routers 232.

[0084] On the other hand, the memory 100 may include one or more data storage fragments 101. The number of routers 232 can be related to the number of data storage fragments 101. For example, the number of routers 232 can be determined based on the number of data storage fragments 101, or conversely, the number of data storage fragments 101 can be determined based on the number of routers 232. For example, the number of routers 232 and the number of data storage fragments 101 can be the same. In this case, a 1:1 correspondence can be formed between routers 232 and data storage fragments 101. In this specification, for ease of understanding and simplified illustration, it is assumed that there are eight routers 232-1 to 232-8. Furthermore, since the data stored in the data storage fragments 101 can be retrieved to the instruction fetch buffer 222 included in the network interface 220, therefore... Figure 3 This indicates that the eight buffers 222-1 to 222-8 are separated from each other. Therefore, each fetch buffer 222 corresponds to a different data storage segment 101, and each fetch buffer 222 can fetch data stored in its corresponding data storage segment 101. However... Figure 3 The examples indicate that the parts are separated for ease of understanding, and are not necessarily limited to physical separation.

[0085] Furthermore, in this specification, when describing data fetching, an example is given where a data packet consists of four flow control units (Flits). Therefore, the interface controller 220 can assign a node ID corresponding to each data memory slice 101 to each flow control unit constituting a data packet. (Refer to...) Figure 3 In the example shown, the four flow control units fetched from fetch buffer 1, 222-1, are assigned node ID "#1". Similarly, the four flow control units fetched from fetch buffer 2, 222-2, are assigned node ID "#2", and the flow control units fetched from the remaining fetch buffers 222-3 to 222-8 are assigned node IDs "#3 to #8".

[0086] According to one embodiment of this specification, the plurality of routers 232 can form a one-dimensional mesh topology at the hardware level. According to another embodiment of this specification, the plurality of routers 232 can also form a two-dimensional mesh topology at the hardware level. The hardware structure of the plurality of routers 232 can be diverse and is not limited to a specific form. Each router 232 can be input to and retrieve data to the instruction fetch buffer 222 and output it to the feed buffer 242, or pass it to other adjacent routers 232. For ease of subsequent explanation, the leftmost router among the plurality of routers will be named sequentially as "Router 1 232-1, Router 2 232-2, ..., Router 8 232-8".

[0087] Figure 4 This is a reference diagram used to illustrate the router structure according to this specification.

[0088] Reference Figure 4 Three routers can be identified. Using router 232-Ref, located in the middle of these three routers, as a reference, the structure of the router according to this specification will be described. Furthermore, the router on the left of the two routers adjacent to the reference router 232-Ref will be named "First Router 232-F," and the router on the right will be named "Second Router 232-S." The terms "first" and "second" are only used to distinguish the two routers adjacent to the reference router 232-Ref and do not indicate any priority order between the routers.

[0089] The router 232 according to this specification may include: a main input port ①, a first transmission output port ②, a first transmission input port ③, a second transmission output port ④, a second transmission input port ⑤, and a main output port ⑥. The main input port ① is the port where data is input to the memory 101, i.e., the instruction fetch buffer 222. The first transmission output port ② is the port for transmitting data to the adjacent first router 232-F. The first transmission input port ③ is the port where data transmitted from the adjacent first router 232-F is input. The second transmission output port ④ is the port for transmitting data to the adjacent second router 232-S. The second transmission input port ⑤ is the port where data transmitted from the adjacent second router 232-S is input. The main output port ⑥ is the port for providing data to the arithmetic unit 300, i.e., the feed buffer 242.

[0090] Therefore, data output through the first transmission output port ② of the reference router 232-Ref is input to the second transmission input port ⑤ of the first router 232-F. Data output through the second transmission output port ④ of the first router 232-F is input to the first transmission input port ③ of the reference router 232-Ref. Data output through the second transmission output port ④ of the reference router 232-Ref is input to the first transmission input port ③ of the second router 232-S. Data output through the first transmission output port ② of the second router 232-S is input to the second transmission input port ⑤ of the reference router 232-Ref.

[0091] On the other hand, refer to Figure 3 Router 1 (232-1) does not show the first transmission output port ② and the first transmission input port ③. Since Router 1 (232-1) may be located on the far left at the physical or software level, the first transmission output port ② and the first transmission input port ③ may not physically exist. Alternatively, Router 1 (232-1) may physically have the first transmission output port ② and the first transmission input port ③, but they may not be used at the software level. Router 8 (232-8) also does not show the second transmission output port ④ and the second transmission input port ⑤ for the same reason as Router 1 (232-1).

[0092] On the other hand, this specification assumes that router 232 transmits data in a counter-clockwise direction. Therefore, when each router 232 transmits input data through the main input port ① and the second transmission input port ⑤, it is assumed that transmission only occurs through the first transmission output port ②. Similarly, when each router 232 transmits input data through the first transmission input port ③, it is assumed that transmission only occurs through the second transmission output port ④. As described above, setting input and output ports during data transmission prevents duplicate data output. Furthermore, it should be understood that the counter-clockwise transmission is not limited to the processing device according to this specification; if the direction is reversed to clockwise, the correspondence between the input and output ports can be adjusted accordingly.

[0093] The router 232 according to this specification can retrieve the node ID of the data input through the main input port ① and process the data with that node ID according to a data processing mapping table. According to one embodiment of this specification, the data processing mapping table can store information about whether to block, reflect, or output the input data. In other words, the router 232 according to this specification can process, according to the data processing mapping table, whether to block, reflect, or output data by not outputting it to other routers based on the node ID. Regarding the data processing mapping table, the router 232 according to this specification can set "outputting data input from one adjacent router to another adjacent router (data transmission)" as a basic operation, but is not necessarily limited to this. Therefore, the data processing mapping table can be considered as information about methods for processing data input from other routers.

[0094] In the data processing mapping table, "blocking" means: preventing data input through the second input port ⑤ or the first input port ③ from being transmitted through the first output port ② or the second output port ④. In the data processing mapping table, "reflection" means: outputting data input through the second input port ⑤ through the second output port ④. Alternatively, in the data processing mapping table, "reflection" means: performing similar processing on data that needs to be output through the first output port ② as on data input through the first input port ③. In the data processing mapping table, "output" means: outputting data input through the first input port ③ through the main output port ⑥.

[0095] Therefore, based on the contents recorded in the data processing mapping table, the software topology composed of multiple routers 232 can present diverse forms. Since the instruction fetching network controller 231 can set whether to perform blocking, reflection, and output according to the topology to be reconstructed, the software topology can be determined by the instruction fetching network controller 231. Regarding the data processing mapping table, through... Figures 5 to 12 Various embodiments will be described in more detail below.

[0096] Figure 5 It is the software topology according to the first embodiment.

[0097] Reference Figure 5 The first embodiment is an example in which data retrieved from each fetch buffer 222 is passed to a feed buffer 242. That is, it is an embodiment in which data stored in fetch buffer 1 222-1 is passed only to feed buffer 1 242-1, and data stored in fetch buffer 2 222-2 is passed only to feed buffer 2 242-2.

[0098] Figure 6 This is a reference diagram of the data processing mapping table according to the first embodiment.

[0099] Reference Figure 6 As can be seen, the data processing mapping table distinguishes data processing methods (e.g., blocking, reflection, output) based on node ID. Furthermore, the items in the data processing mapping table can record whether a corresponding operation is executed. A record of "1" indicates that the item is executed, and a record of "0" indicates that the item is not executed.

[0100] According to the data processing mapping table, observe Figure 5 Router 232-1 (number 1) does not block data with ID#1, but performs reflection and output. Since this specification assumes that router 232 transmits data counter-clockwise, the ID#1 data input through main input port ① of router 232-1 will be transmitted through the first transmission output port ②. Because this specification assumes that router 232 transmits data counter-clockwise, the ID#1 data input to main input port ① of router 232-1 is transmitted through the first transmission output port ②. At this time, since reflection of ID#1 data is configured in the data processing mapping table of router 232-1, the ID#1 data that needs to be output through the first transmission output port ② undergoes similar processing as the data input through the first transmission input port ③. Furthermore, since output of ID#1 data is configured in the data processing mapping table of router 232-1, the ID#1 data is output through the main output port ⑥. For the remaining ID#2~#8 data, blocking is performed, and neither reflection nor output is performed. Therefore, the data retrieved... Figure 5 Data in fetch buffer 222-1 can be output only to feed buffer 242-1. Figure 5 Router 232-2 to router 832-8 also operate on the same principle, so unnecessary repetition will be omitted.

[0101] Figure 7 It is the software topology according to the second embodiment.

[0102] Reference Figure 7 The second embodiment is an example in which data retrieved from each fetch buffer 222 is passed to two feed buffers 242. That is, it is an embodiment in which data stored in fetch buffer 1 222-1 is passed to feed buffer 1 242-1 and feed buffer 242-2, and data stored in fetch buffer 2 222-2 is passed to feed buffer 1 242-1 and feed buffer 242-2.

[0103] Figure 8 This is a reference diagram of a data processing mapping table according to the second embodiment.

[0104] According to the data processing mapping table, observe Figure 7 Router 232-1 (number 1) does not block ID#1 data, but instead performs reflection and output. Since the method for outputting ID#1 data to feed buffer 242-1 (number 1) has already been explained in the first embodiment, a repeating explanation is omitted. Furthermore, because ID#1 data undergoes similar processing to data input through the first transmission input port ③, ID#1 data can be transmitted to router 232-2 (number 2) through the second transmission output port ④. (Observation) Figure 7 Router 2 (232-2) does not block or reflect data with ID#1, but instead outputs it. Therefore, if router 232-2 receives data with ID#1 from router 1 (232-1), it can output it to feed buffer 242-2. Thus, the data retrieved... Figure 7 The data in fetch buffer 222-1 can be output to feed buffer 242-1 and feed buffer 242-2.

[0105] If we observe router 232-2, the ID#2 data is neither blocked nor reflected; instead, it is output. Since this specification assumes router 232 transmits data counter-clockwise, the ID#2 data input to the main input port ① of router 232-2 can be transmitted to router 1 232-1 through the first transmission / output port ②. Furthermore, if we observe… Figure 7 Router 1 (232-1) does not block the data with ID#2, but reflects and outputs it. The ID#2 data input to Router 1 (232-1), being in the same location as the ID#1 data, can be output to Feedback Buffer 1 (242-1) via Router 1 (232-1). Furthermore, the ID#2 data can be transmitted again to Feedback Buffer 242-2. Router 2 (232-2) can output the ID#2 data, which was input again from Router 1 (232-1) via the first input port ③, through the main output port ⑥. Therefore, the data retrieved from... Figure 7The data in fetch buffer 222-2 can be output to feedback buffer 1 242-1 and feedback buffer 242-2.

[0106] On the other hand, Figure 7 Since the ID#1 data is input through the first transmission input port ③ of router 232-2, it is output through the second transmission output port ④ of router 232-2. Therefore, the ID#1 data will not be input to router 1232-1 again. Furthermore, router 32-3 blocks the input of ID#1 data through its own first transmission input port ③. Additionally, router 232-3 also blocks the input of ID#2 data through its own first transmission input port ③.

[0107] Figure 7 Router 1 (232-1) and Router 2 (232-2) block the remaining data with IDs #3 to #8, and neither reflect nor output it.

[0108] because Figure 7 Router 3 (232-3), router 4 (232-4), router 5 (232-5), router 6 (232-6), router 7 (232-7), and router 8 (232-8) also operate on the same principle, so unnecessary repetition will be omitted.

[0109] Figure 9 It is a software topology based on the third embodiment.

[0110] Reference Figure 9 The third embodiment is an example in which data retrieved from each fetch buffer 222 is passed to four feedback buffers 242. That is, it is an embodiment in which data stored in fetch buffers 222-1 to 222-4 (numbers 1 to 4) is passed to feedback buffers 242-1 to 242-4 (numbers 1 to 4), respectively.

[0111] Figure 10 This is a reference diagram of the data processing mapping table according to the third embodiment.

[0112] Since the processing of ID#1 data and ID#2 data has been previously described through the first and second embodiments, therefore... Figure 10 The lieutenant general representatively explained the extraction to Figure 9 The processing of ID#3 data in instruction fetch buffer 222-3. First, the ID#3 data fetched into instruction fetch buffer 222-3 is input through the main input port ① of router 3 232-3, and output through the first pass-out port ② of router 3 232-3.

[0113] Router 232-2 receives ID#3 data through its second transmission input port ⑤ and outputs ID#3 data through its first transmission output port ②.

[0114] Router 232-1 receives ID#3 data through its second pass-in input port ⑤. According to the data processing mapping table of Router 232-1, since Router 232-1 performs reflection and output on ID#3 data, ID#3 data is output to Feedback Buffer 242-1 through the main output port ⑥, and also output through the second pass-out output port ④ of Router 232-1.

[0115] Router 232-2 receives ID#3 data through its first transmission input port ③. According to the data processing mapping table of Router 232-2, since Router 232-2 performs output on ID#3 data, ID#3 data is output to Feedback Buffer 242-2 through the main output port ⑥, and simultaneously output through the second transmission output port ④ of Router 232-2.

[0116] Router 232-3 receives ID#3 data through its first transmission input port ③. According to the data processing mapping table of router 232-3, since router 232-3 performs output on ID#3 data, the ID#3 data is output to feedback buffer 242-3 through the main output port ⑥, and simultaneously output through the second transmission output port ④ of router 232-3.

[0117] Router 4 (232-4) receives the ID#3 data through its first transmission input port ③. According to the data processing mapping table of router 4 (232-4), since router 4 (232-4) performs output on the ID#3 data, the ID#3 data is output to the feedback buffer 242-4 (242-4) through the main output port ⑥, and simultaneously output through the second transmission output port ④ of router 4 (232-4).

[0118] Router 5 (232-5) receives ID#3 data through its first transmission input port ③. According to the data processing mapping table of router 5 (232-5), since router 5 (232-5) blocks ID#3 data, ID#3 data will no longer be output or transmitted.

[0119] therefore, Figure 9The ID#3 data retrieved from fetch buffer 222-3 (number 3) can be passed to feedback buffers 242-1 to 242-4 (numbers 1 to 4). Similarly, ID#1, ID#2, and ID#4 data can also be passed to feedback buffers 242-1 to 242-4 (numbers 1 to 4). On the other hand, ID#5 to ID#8 data can be passed to feedback buffers 242-5 to 242-8 (numbers 5 to 8) according to the same principle.

[0120] Figure 11 It is the software topology according to the fourth embodiment.

[0121] Reference Figure 11 The fourth embodiment is an example in which data retrieved from each fetch buffer 222 is passed to all feedback buffers 242. That is, it is an embodiment in which data stored in fetch buffers 222-1 to 222-8 (numbers 1 to 8) is passed to feedback buffers 242-1 to 242-8 (numbers 1 to 8), respectively.

[0122] Figure 12 This is a reference diagram of the data processing mapping table according to the fourth embodiment.

[0123] Previously, since the first to third embodiments have fully explained how each router 232 processes the input data according to the data processing mapping table, repeated explanations are omitted.

[0124] like Figures 5 to 12 As shown, the instruction fetching network controller 231 can make the same settings for blocking and output in the data processing mapping table of routers 232 belonging to the same group within the reorganized software topology.

[0125] On the other hand, this concludes the explanation of how router 232 processes a single piece of data. However, multiple pieces of data fetched into multiple fetch buffers 222 need to be processed collaboratively. Existing technology allows routers to have sufficiently large buffers and to resolve conflicts. However, it is obvious that excessively large buffers can cause various problems such as increased processor size, increased power consumption, increased heat generation, and processing latency. While it is well known that having appropriately sized buffers is optimal, if the buffers included in any router are full, the problem arises where data input / output to all connected routers needs to be delayed. Furthermore, when the latency between routers is variable (e.g., in cases where clock domains differ), more complex problems may occur. Therefore, a method is needed that can properly control the timing of inputting the fetched data to each router 232. According to the arithmetic processing apparatus 10 of this specification, a more efficient method for processing multiple pieces of data can be proposed.

[0126] Figure 13 This is an example diagram of data input and output control according to one embodiment of this specification.

[0127] Reference Figure 13 It can be confirmed that the software topology of multiple routers 232 is consistent with... Figure 9 The third embodiment shown is the same. Therefore, when data is fetched from instruction fetch buffers 1 to 4 (222-1 to 222-4) and input to routers 1 to 4 (232-1 to 232-4), it must be output to feed buffers 1 to 4 (242-1 to 242-4) without causing conflicts. On the other hand, although the control of data input and output is described in this specification through the third embodiment, the arithmetic processing device 10 according to this specification is not limited to the example described.

[0128] exist Figure 13 The example shown illustrates how a data packet can be composed of two flow control units (Flits). Furthermore, in... Figure 13 In the example shown, the numbers marked on the flow control unit indicate the order in which the inputs are fed to the feed buffer 242.

[0129] In this specification, the data input from memory 100 to router 232 may include tail information and tail maintenance information. For example, the data flit fetched to the instruction fetch buffer 222 may have a format including header, payload, tail maintenance information (keep_tail), and tail information. Figure 13The flow control unit shown is in the format of "payload (P)" followed by "tail maintenance information (keep_tail)" and "tail information (tail)".

[0130] The tail information can be either a first logical value T representing the end of a data packet or a second logical value F representing the non-end of a data packet. The tail maintenance information can be either a first logical value T representing the maintenance of the tail information or a second logical value F representing the modification of the tail information.

[0131] According to one embodiment of this specification, the interface controller 221 can further set the tail information and the tail maintenance information in the retrieved data.

[0132] According to another embodiment of this specification, the network interface 220 may include: independent structural elements for setting the tail information and the tail maintenance information in the retrieved data.

[0133] The interface controller 221 can set the tail information of the last data packet in the extracted data packet to a first logical value T, and set the tail information of the remaining data to a second logical value F.

[0134] Furthermore, the interface controller 221 can set the tail maintenance information of the data with the last node ID in the software topology of the router constructed by the instruction fetching network controller 231 through the data processing mapping table to a first logical value T representing the maintenance of tail information, and set the tail maintenance information of the data with the remaining node IDs to a second logical value F representing the modification of tail information.

[0135] Based on the above settings, observe Figure 13 It can be confirmed that among the data retrieved into each buffer 222, the tail information of the flow control units with even-numbered order is "F", and the tail information of the flow control units with odd-numbered order is "T". Furthermore, it can be confirmed that only the tail maintenance information of buffer 4 (222-4) and buffer 8 (222-8), which have the last node ID of the software topology structure, is "T".

[0136] The router 232 according to this specification may include an arbiter 233 that controls the input port based on the trailer information of the input data and modifies the trailer information based on the trailer maintenance information of the input data. The operation of the arbiter 233 will be described in more detail below.

[0137] Figures 14 to 20 This is a reference diagram of the data processing in the arbitration department.

[0138] Figures 14 to 20For the sake of simplicity, only instruction fetch buffers 222-1 to 222-4 (numbers 1 to 4) and routers 232-1 to 232-4 (numbers 1 to 4) are shown.

[0139] Reference Figure 14 First, the arbitrator unit 233 can be set to an initial state such that data is input from the memory 100, i.e., from the main input port ①, in the port related to the input / output of the router 232. Based on the initial state, the two flow control units retrieved from each buffer 222 can be sequentially input to each router 232.

[0140] When the arbiter unit 233 controls the input port to input data from the memory 100, if the tail information of the data input from the memory 100, i.e., the instruction fetch buffer 222, is a first logic value T, the input port can be changed to input data from an adjacent router.

[0141] Reference Figure 15 Since the tail information of the second flow control unit input to each router is "T", it can be confirmed that the above-mentioned arbiter unit has been changed to input data from the main input port ① of each router to the second transmission input port ⑤.

[0142] On the other hand, when the tail information of the data input from the memory (main input port) by the aforementioned arbitrator is maintained at the second logic value F, the tail information of the input data can be changed to the second logic value F. Figure 13 In the example shown, since the tail information of flow control units F0 to F5 is all "F", the tail information of flow control units F0 to F5, especially the tail information of flow control units F1, F3, and F5, is changed to "F". On the other hand, since the tail information of flow control units F6 and F7 is all "T", the tail information of flow control units F6 and F7 is not changed. In particular, it should be noted that the tail information of flow control unit F6 remains "F", while the tail information of flow control unit F7 remains "T".

[0143] Furthermore, data with retained or modified trailing information can be output to neighboring routers. When the trailing information of the data input from the neighboring router by the arbitrator unit 233 is a first logic value T, it can control the input of data from the aforementioned memory. (Refer to...) Figure 16 This confirms that the data has been transmitted to the adjacent routers. In particular, observe router 3, 232-3. Router 3, 232-3, receives the inputs from Fleet F6 and F7 through the second transmission input port ⑤. Since the trailing information of Fleet F7 is "T", it can control the input port, allowing data to be input from the main input port ①.

[0144] On the other hand, as with router 4 232-4, when configured not to input data from the aforementioned adjacent routers, the arbitration unit 233 can control the input of data from the main input port ① when the tail information of the data output to the adjacent router through its own first transmission output port ② is the first logic value T.

[0145] After that, observation Figures 17 to 20 This allows us to confirm the sequential transmission of data. In particular, if we observe... Figure 20 The data output from router 232-1 (number 1) confirms that the output order matches the order of the flow control units. Furthermore, it confirms that... Figure 20 The status of flow control units 8 to 15 and in Figure 14 The initial states mentioned are very similar. Therefore, it can be predicted that flow control units 8 to 15 will also be transmitted sequentially.

[0146] According to one aspect of this specification, the data flow control logic is simplified, thereby eliminating or minimizing router buffers. This reduces the area occupied by the router within the chip and lowers power consumption. According to another aspect of this specification, the packet information used for multicast can be minimized. According to yet another aspect of this specification, even with an increase in the number of nodes, the increase in cost can be minimized, and maximum bandwidth can be achieved.

[0147] The embodiments of this specification have been described above with reference to the accompanying drawings. However, those skilled in the art will understand that the invention can be implemented in other specific forms without departing from the technical concept or essential characteristics of the invention. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

[0148] Explanation of reference numerals in the attached figures:

[0149] 10: Computational processing unit

[0150] 100: Memory

[0151] 101: Data Storage Sharding

[0152] 200: Instruction Fetch Unit

[0153] 210: Instruction Fetch Sequencer

[0154] 220: Network Interface

[0155] 221: Interface Controller

[0156] 222: Instruction Fetch Buffer

[0157] 230: Pointer fetching network

[0158] 231: Instruction Fetch Network Controller

[0159] 232: Router

[0160] 233: Arbitration Department

[0161] 240: Feeding Module

[0162] 242: Feed Buffer

[0163] 250: Operation Sequencer Module

[0164] 300: Arithmetic Unit

[0165] 310: Dot Product Engine

[0166] 400: Submission Unit

[0167] 410: Submit sequencer

Claims

1. An arithmetic processing device, comprising an instruction fetch unit, wherein the instruction fetch unit reads data required for computation from memory and provides it to an arithmetic unit to perform neural network processing. The instruction fetching unit includes: Multiple routers have a data processing mapping table, which records how the input data is processed according to the node ID of the input data; as well as The network controller is used to reconstruct the data processing mapping tables of the multiple routers to form a software topology based on the computational format. Each of the routers includes an arbitration unit, which controls the input port based on the tail information of the input data and modifies the tail information based on the tail maintenance information of the input data.

2. The computing processing device according to claim 1, characterized in that, Each of the routers mentioned includes: The system includes a main input port for inputting data from the memory, a first transmission output port for transmitting data to a neighboring first router, a first transmission input port for inputting data transmitted from a neighboring first router, a second transmission output port for transmitting data to a neighboring second router, a second transmission input port for inputting data transmitted from a neighboring second router, and a main output port for providing data to the computing unit.

3. The computing processing device according to claim 1, characterized in that, The data processing mapping table is used to store information about whether to perform input data blocking, reflection, and output.

4. The computing processing device according to claim 3, characterized in that, The instruction fetching network controller is used to set whether to perform the blocking, reflection, and output based on the topology to be reconstructed.

5. The arithmetic processing device according to claim 4, characterized in that, The instruction fetching network controller is used to set the blocking and output in the data processing mapping table of routers belonging to the same group within the reconstructed topology to be the same.

6. The arithmetic processing device according to claim 1, characterized in that, The instruction fetching unit includes: A network interface for retrieving data stored in the memory; and The fetch network is used to transmit data retrieved to the network interface. The plurality of routers and instruction fetching network controllers are included in the instruction fetching network.

7. The arithmetic processing apparatus according to claim 6, characterized in that, The network interface includes: The fetch buffer is used to store the fetched data; and An interface controller is used to assign the retrieved data to a node ID corresponding to each data storage shard.

8. The arithmetic processing device according to claim 1, characterized in that, The tail information is either a first logical value T representing the end of the data packet or a second logical value F representing the end of a data packet. The tail maintenance information is either a first logic value T representing the maintenance of tail information or a second logic value F representing the change of tail information.

9. The arithmetic processing device according to claim 8, characterized in that, The arbitration unit is configured to, when controlling the input port to input data from the memory, if the tail information of the input data from the memory is a first logic value T, change the input port to input data from a neighboring router.

10. The arithmetic processing apparatus according to claim 8, characterized in that, The arbitration unit is used to change the tail information of the input data to the second logic value F when the tail information of the input data from the memory is maintained at the second logic value F.

11. The arithmetic processing apparatus according to claim 8, characterized in that, The arbitration unit is used to control the input data from the memory when the tail information of the input data from the adjacent router is a first logic value T.

12. The arithmetic processing apparatus according to claim 7, characterized in that, The interface controller is used to further set the tail information and the tail maintenance information on the retrieved data.

13. The arithmetic processing apparatus according to claim 12, characterized in that, The interface controller is used to set the tail information of the last data in the extracted data packet to a first logical value T, and set the tail information of the remaining data to a second logical value F.

14. The arithmetic processing apparatus according to claim 12, characterized in that, The interface controller is configured to, in the software topology of the router constructed by the instruction fetching network controller based on the data processing mapping table, set the tail maintenance information of the data with the last node ID to a first logical value T representing the maintenance of tail information, and set the tail maintenance information of the data with the remaining node IDs to a second logical value F representing the change of tail information.

Images