In-network compute engine for network switches

Abstract

This disclosure relates to an in-network computing engine for a network switch. A data plane of a network switch may include: a packet switch including a first port and a second port; a first processor coupled to the packet switch and a control plane, the first processor including: a first digital circuit configured to: receive a first payload of a first packet ingress into the first port; and convert first numerical data in the first payload from a first format to a second format; and a second digital circuit configured to: receive the first numerical data and the second numerical data from the first digital circuit; and perform a first computation operation using the first and second numerical data as input; wherein the first processor is configured to return a second packet to the packet switch for export from the second port, the second payload of the second packet containing at least partially numerical data generated by the first computation operation.

Description

Technical Field

[0001] This disclosure relates to an in-network computing engine for network switches. Background Technology

[0002] Artificial intelligence (AI) can be the scientific field involving building machines that can reason, learn, and act in ways that typically require human intelligence or involve data on scales exceeding human analytical capabilities. Machine learning (ML) enables machines to learn and improve autonomously without being explicitly programmed. Deep learning can be a type of machine learning that uses artificial neural networks to process and analyze information. This paper describes various examples in the context of machine learning, which may include deep learning, but is not intended to exclude any other forms of AI.

[0003] ML models can be a set of algorithms used in machine learning. For example, an ML model can be trained on data to produce an estimate of the data pattern. The data pattern estimate can be used to make predictions, classify, etc., based on input data. In machine learning, training can be the process of evaluating the data pattern estimate against a known data pattern using an error function and adjusting the model's parameters to minimize the error function. Distributed learning enables a system (e.g., multiple machines) to implement an ML model by dividing it into local and global models. In a distributed learning system, training data can be distributed from a central machine to multiple local machines. The training data can be used to train local models on the local machines, and the training results can be aggregated and used to update the global model at the central machine. Distributed learning can use data parallelism (e.g., for distributing different datasets across devices, for example, for training or for inference), model parallelism (e.g., for distributing different parts of the ML model across devices), or both.

[0004] A computer can be a machine used in a distributed learning system to implement local and global models and process data. The computer may contain dedicated hardware for computation, referred to as a hardware accelerator. Hardware accelerators can be, for example, graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and the like, and may contain circuitry optimized for the type of computation used in the ML model. The computer can be part of a network and exchange information, such as computation results. For example, some computers may use their hardware accelerators (e.g., local models) to generate intermediate computation results, and these intermediate computation results can be sent over the network to another computer for aggregation (e.g., at the global model). The aggregation of intermediate computation results (also referred to as the reduction of intermediate computation results) can be performed by the computer's central processing unit (CPU) or hardware accelerator. In this scenario, the network is only used to transmit intermediate computation results. Then, before using the aggregated and reduced results (e.g., updating the global model), the receiving computer must expend resources to receive the intermediate results, convert the format of the results, aggregate and reduce the results, and the like. Summary of the Invention

[0005] In some embodiments, the data plane of a network switch may include a packet switch. The packet switch may include a first port and a second port. The data plane may include a first processor coupled to the packet switch and the control plane of the network switch. The first processor may include a first digital circuit configured to receive a first payload of a first packet imported to the first port and to convert first numerical data in the first payload from a first format to a second format. The first processor may include a second digital circuit configured to receive the first and second numerical data from the first digital circuit and to perform a first computation operation using the first and second numerical data as input. The first processor may be configured to return a second packet to the packet switch for export from the second port. The second payload of the second packet may include at least partially numerical data generated by the first computation operation.

[0006] In some embodiments, a network switch may include a control plane including a central processing unit (CPU) and memory. The network switch may include a data plane. The data plane may include a packet switch and a first processor. The packet switch may include a first port and a second port. The first processor may be coupled to the packet switch and the control plane. The first processor may include a first digital circuit configured to receive a first payload of a first packet imported to the first port and to convert first numerical data in the first payload from a first format to a second format. The first processor may include a second digital circuit configured to receive the first and second numerical data from the first digital circuit and to perform a first computation operation using the first and second numerical data as input. The first processor may be configured to return a second packet to the packet switch for export from the second port. The second payload of the second packet may include numerical data at least partially generated by the first computation operation. The second format and the first computation operation can be selected by configuration of the first processor by the control plane.

[0007] In some embodiments, a method for processing packets in a network switch may include receiving a first payload of a first packet introduced into a first port of the packet switch at a first digital circuit of a first processor coupled to a packet switch and the control plane of the network switch. The method may include converting the first numerical data in the first payload from a first format to a second format by the first digital circuit. The method may include receiving the first numerical data and second numerical data from the first digital circuit at a second digital circuit. The method may include performing a first calculation operation by the second digital circuit using the first and second numerical data as input. The method may include returning a second packet from the first processor to the packet switch for export from a second port of the packet switch. The second payload of the second packet may contain numerical data at least partially generated by the first calculation operation. Attached Figure Description

[0008] Figure 1 It is a block diagram depicting a network according to some embodiments.

[0009] Figure 2 This is a block diagram depicting a network switch according to some embodiments.

[0010] Figure 3A This is a block diagram depicting an in-network computing engine (ICE) according to some embodiments.

[0011] Figure 3BIt is a block diagram depicting groups that can be processed by ICE examples according to some embodiments.

[0012] Figure 4 This is a flowchart depicting a method for processing packets in a network switch according to some embodiments.

[0013] Figure 5 This is a block diagram depicting the computing unit of an ICE according to some embodiments.

[0014] Figure 6 It is a block diagram depicting computing elements according to some embodiments.

[0015] Figure 7 According to some embodiments Figure 6 A block diagram of digital logic in computing elements.

[0016] Figure 8 This is a block diagram depicting a Unified Multi-Format (UMF) adder according to some embodiments.

[0017] Figure 9 This is a block diagram depicting a UMF multiplier according to some embodiments.

[0018] Figure 10A It is a table summarizing the data processing performed by a UMF adder or UMF multiplier according to some embodiments.

[0019] Figure 10B This is a table summarizing the data processing performed by the UMF adder or UMF multiplier according to other embodiments.

[0020] Figure 11 This is a block diagram depicting a conversion unit of an ICE according to some embodiments.

[0021] Figure 12 This is a summary based on some embodiments. Figure 11 The table of operations for the conversion unit.

[0022] Figure 13 This is a block diagram depicting a UMF converter according to some embodiments.

[0023] Figure 14 It is a table that summarizes the format conversion of elements based on some embodiments.

[0024] Figure 15A This is a block diagram depicting an example of an ICE according to some embodiments.

[0025] Figure 15B This is a block diagram depicting another example of an ICE according to some embodiments.

[0026] Figure 15CThis is a block diagram depicting another example of an ICE according to some embodiments.

[0027] Figure 16A This is a block diagram depicting an ICE cluster according to some embodiments.

[0028] Figure 16B This is a block diagram depicting another ICE cluster according to some embodiments.

[0029] Figure 16C This is a block diagram depicting another ICE cluster according to some embodiments.

[0030] Figure 16D This is a block diagram depicting another ICE cluster according to some embodiments. Detailed Implementation

[0031] Figure 1 This is a block diagram depicting a network 100 according to some embodiments. The network may be a combination of devices and their connecting links. The devices in network 100 may include computers 120…12 n (where n is an integer greater than 1) and a network switch 10. The computer may be a device capable of storing and processing digital data. The network switch may be a device (e.g., a dedicated computer) that uses a switching connection network to receive and forward data from other devices (e.g., other computers). Computers 120…12 n It can be coupled to network switch 10 via wired, wireless, fiber optic, or similar connection links known in the field. Computer 120…12 n Executable software, including ML software 140…14 n Software can be instructions that can be programmed to manipulate hardware (e.g., a processor) to operate on digital data. ML software can be software used to implement ML models.

[0032] Digital data can be numbers represented by a finite number of bits. A computer can represent digital data as binary numbers represented by a finite number of digits (bits). Digital data can have different interpretations. Control data can be digital data used to control, configure, manipulate, etc., a system or its components (e.g., a machine or software). Input data can be digital data input to a system or its components. Output data can be digital data output by a system or its components. Packet data can be digital data in a packet. Payload data can be digital data in the payload of a packet. Header data can be digital data in the header of a packet. Numerical data can be digital data representing numbers. Operands can be numerical data input to an operation. Operation result (“result”) can be numerical data produced by an operation on operands. Numerical data can have a format. The format of numerical data can be a way of representing numbers. The format can specify the number of bits representing the number. The format can specify the type of number, such as integer and floating-point number. The format can specify the precision of the number (e.g., how many digits are on either side of the decimal point). Other types of digital data may be defined below with reference to embodiments.

[0033] In some embodiments, computer 12 k (k ML in {1, 2, …, n} k Intermediate computation results can be generated. In the context of ML models, intermediate computation results can be numerical data generated from partial computations. For example, ML... k This can be a local model in a distributed learning system. A local model uses only a portion of the training data, a portion of the inference data, a portion of the ML model, etc., and therefore the computational output of the local model is an intermediate computation result. ML0 in computer 120 can be configured to receive data from computers 121…12. n The aggregated computation results. For example, ML0 can implement a global model of a distributed learning system. The aggregated computation results can be numerical data generated by mathematical combinations of intermediate computation results. For example, ML0 can use the aggregated computation results to update the global model. ML0 in computer 120 can also send the reduced results or the updated global model back to computer 120. k For ML k deal with.

[0034] In some embodiments, network switch 10 may include an in-network computing engine (ICE) 16. The processor may be circuitry capable of processing digital data. The processor may be dedicated (e.g., may include digital circuitry implementing fixed-function logic) rather than general-purpose (e.g., a processor capable of executing software in a fetch-decode-execute model). The digital circuitry may be circuitry that processes digital signals transmitting digital data. The ICE may be a dedicated processor and may process packets containing numerical data transmitted from devices on the network within the network switch. The ICE 16 may perform calculations on the numerical data in packets received by network switch 10. Embodiments of the ICE 16 and its digital circuitry are described below.

[0035] Computer 12 k (k ML in {1, 2, …, n} k Intermediate computation results can be sent from network switch 10 to computer 120 performing ML0. Computer 120 can execute configuration software (“Configuration 15”) that interacts with network switch 10 and provides control data for configuring ICE 16. Network switch 10 can forward intermediate computation results to ICE 16 as configured for processing, which may include aggregating and reducing the intermediate computation results to aggregated and reduced computation results. Then, network switch 10 can forward the aggregated and reduced computation results to computer 120 for ML0 processing. Thus, intermediate computation results generated by the local model of the distributed learning system can be aggregated and reduced within the network (e.g., by network switch 10) during transmission to the destination (e.g., the global model). This eliminates the need to perform the aggregation and reduction process on the global model (e.g., ML0 in computer 120) itself, thereby saving resources of computer 120. In some embodiments, network switch 10 can forward and broadcast aggregated and reduced computation results back to one or more computers 120. k For ML k deal with.

[0036] To understand in-network computation operations in ML applications using ICE 16, refer to the following example. An example ML model can be a neural network (also known as an artificial neural network), which can be a model of linked processing nodes that simulate the functions of the human brain. A neural network can contain layers of nodes with input layers, one or more hidden layers, and an output layer. Each node (e.g., an artificial neuron) can be connected to at least one other node, and the connections between nodes can have weights. Weights determine the strength of the connections between nodes. A node can receive one or more inputs (e.g., from weighted connections), perform computations, and produce outputs. A node can apply an activation function to the computation, and the node's output can be considered its activation. Activations can be passed to other nodes through weighted connections. Nodes can also have biases that can adjust the threshold of the activation function. Weights and biases can be parameters of a model including the neural network. Training a neural network can involve updating weights and biases to minimize a loss function using gradients. The gradient can be a measure of the change in parameters relative to a function of parameters. In mathematical terms, the gradient can be calculated using the partial derivatives of a function with respect to its parameters. For example, for a function f(θ) (where θ represents the parameters of the machine learning model), the gradient... It can be a vector containing the partial derivatives of f with respect to each parameter in θ. The function f can be a loss function. A loss function can be a function that measures the difference (e.g., error) between the output (also known as the predicted output) of a machine learning model and the target output (also known as the actual output).

[0037] In some embodiments, a distributed learning system may use data parallelism, model parallelism, or both. (Computer 12) k (k ML in {1, 2, …, n} k Each ML0 can be a local model that includes all or a part of the neural network. ML0 in computer 120 can be a global model that includes the neural network. For training, ML0 can send training data to each ML0. k Each ML k Training can be performed to generate gradients and update the parameters of the corresponding local model. Gradients can be intermediate calculations because they are generated using only a portion of the training data and / or a portion of the model. Updated parameters can also be intermediate calculations because they are trained using only a portion of the training data and / or a portion of the model. Each ML... k Gradient / updated parameters can be sent from network switch 10 to ML0 in computer 120. Configuration 15 can send control data to network switch 10 to configure ICE 16 aggregation and reduction from computers 121…12. nReceived intermediate computation results. For example, ICE 16 can aggregate and reduce gradients or parameters as configured to generate aggregated and reduced computation results. ICE 16 can forward the aggregated and reduced gradients / parameters to ML0 as configured, ML0 can be used to update the global model. The above process is described with respect to backpropagation in training with data parallelism. Using model parallelism, similar aggregation and reduction can be performed for forward propagation during training or for inference. Local models can locally generate intermediate computation results. ICE 16 can be configured to aggregate and reduce local computation results to generate aggregated and reduced computation results, which can then be forwarded to a destination. In some embodiments, network switch 10 can forward and broadcast aggregated and reduced computation results back to one or more computers 12. k For ML k deal with.

[0038] Figure 2 This is a block diagram depicting a network switch 10 according to some embodiments. The network switch 10 may include a control plane 20 and a data plane 21. The data plane of the network switch may be circuitry for processing and forwarding network traffic (“traffic”). Traffic may be network packets transmitted over time. Network packets may be data units transmitted in a network using one or more protocols. For example, network packets may be transmitted by devices on the network using layered protocols of the Open Systems Interconnection (OSI) model, which is well known in the art. Network packets may have variable lengths (e.g., variable number of bytes). The data plane 21 may process network packets into smaller, fixed-size data units for internal communication within its circuitry (referred to as cells). Cells may be data units transmitted within the network switch. As used herein, a packet may be a network packet or a cell. A packet may include a header and a payload. The payload of a packet may be any digital data. The header of a packet may be control data, such as protocol control data. The header may include stack or multi-level headers. The control plane of a network switch can be circuitry used to control the data plane (e.g., building and maintaining forwarding and routing tables used by the data plane).

[0039] Control plane 20 may include a central processing unit (CPU) 24 and memory 26. The CPU may be a device programmable by instructions to process digital data. The memory may be a device capable of storing digital data. CPU 24 may execute software stored in memory 26 to control data plane 21. Data plane 21 may include packet switch 28 and one or more ICEs 16. Packet switch 28 may be circuitry configured to process and forward network packets. Packet switch 28 may include, for example, physical interfaces (physical ports) having transceivers for transmitting and receiving electrical signals, ingress logic for processing incoming network packets (e.g., including packet-to-cell segmentation), switching structures for transmitting packets within the physical port, memory for storing packets, egress logic for processing outgoing network packets (e.g., including cell-to-packet reassembly), and the like, each of which is known in the art. Packet switch 28 may be coupled to ICE 16. Packet switch 28 and ICE 16 may be circuitry of integrated circuits (ICs), such as application-specific integrated circuits (ASICs) 22. ICs may be electronic devices formed on a semiconductor substrate. An ASIC is an IC that can be customized for a specific function. ASIC 22 can be customized for a specific function performed by packet switch 28 and ICE 16. ASIC 22, CPU 24, and memory 26 can be electrically and mechanically mounted to one or more printed circuit boards (PCBs) 18. The PCB can be a well-known structure used for mechanically fixing and electrically connecting electronic devices. PCB 18 may include connectors 30 coupled to ASIC 22. ASIC 22 can send signals to and receive signals from the network through connectors 30.

[0040] In operation, devices on the network can send packets along with control data to control plane 20 for the purpose of configuring data plane 21. This configuration may include the configuration of ICE 16. ICE 16 may have various configurable aspects, and the configured ICE 16 may be referred to as an ICE example. This configuration may include associating a set of ports with ICE 16. Ports may be interfaces identified by network switches as ingress points, egress points, or both for network packets. Ingress traffic may be traffic received from another device at a port of the network switch. Egress traffic may be traffic transmitted from a port of the network switch to another device. Ports may be physical ports (e.g., physical interfaces of the network switch) or logical (virtual) ports. Logical (or virtual) ports may be software-defined interfaces of the network switch (e.g., defined by control plane 20). Logical ports may include, for example, Virtual Local Area Network (VLAN) interfaces, Link Aggregation Groups (LAGs), and the like, each known in its respective domain. Packet switch 28 may forward some of the network traffic received from the ports to the ICE example. The ICE example can process numerical data in ingress traffic (e.g., aggregation and reduction of intermediate calculation results, as discussed above). The ICE example can return network traffic to packet switch 28 for forwarding to other devices in the port, where the numerical data of the returned traffic may contain aggregated and reduced calculation results.

[0041] In some embodiments, ICE 16 can be configured to manipulate packet processing in accordance with packet processing performed by packet switch 28. For example, packet switch 28 may receive two network packets at two ports. ICE 16 can be configured such that packet processing by packet switch 28 passes the two network packets to ICE 16, ICE 16 processes the two packets and generates a result packet, which is then passed back to packet processing by packet switch 28. Furthermore, ICE 16 can improve the underlying technology of the distributed learning system by aggregating and reducing intermediate computation results within the network at the network switch, thereby eliminating the need for the destination to expend resources on aggregation and reduction.

[0042] Figure 3AThis is a block diagram depicting an ICE 16 according to some embodiments. The ICE 16, packet switch 28, and control plane 20 are shown in the context. The ICE 16 may include interfaces 3051…3054. Interfaces of the device may be input points, output points, or both. Interfaces 3051 and 3052 may be input points (input interfaces) to the ICE 16. Interfaces 3053 and 3054 may be output points (output interfaces) from the ICE 16. The control plane 20 may associate ports of the packet switch 28 with interfaces 3051…3053 and optionally interface 3054. In an example, port 3021 may be associated with input interface 3051, port 3022 may be associated with input interface 3052, and port 3023 may be associated with output interface 3053. ICE 16 can receive a first packet of data (referred to as IN_DATA0) imported from packet switch 28 to port 3021, and a second packet of data (also referred to as IN_DATA1) imported from packet switch 28 to port 3022. ICE 16 can supply packet data to packet switch 28 for export from port 3023 (referred to as OUT_DATA). ICE 16 can use the first numerical data in IN_DATA0 and the second numerical data in IN_DATA1 as operands to perform calculations to generate a third numerical data as the result in OUT_DATA. The packet data imported to port 3021 can come from a device on the network (e.g., a computer supplying intermediate calculation results) or from another ICE instance in the ICE cluster (discussed further below). Similarly, the packet data imported to port 3022 can come from a device on the network or from another ICE instance in the ICE cluster. Port 3023 can transmit packet data to a device on the network (e.g., a computer collecting aggregated calculation results). In some configurations, control plane 20 can associate output interface 3053 with multiple ports 302. ICE 16 can supply packet data to packet switch 28, and packet switch 28 can broadcast packet data to be exported from multiple ports, each of which transmits packet data to devices on the network.

[0043] In some configurations, control plane 20 can associate output interface 3054 with port 3024. ICE 16 can supply packet data to packet switch 28 for transmission via port 3024 (referred to as R_DATA). The fourth numerical data in R_DATA can be the result of intermediate calculations performed by ICE 16 (referred to as reduced data), as discussed further below. Port 3024 can transmit packet data to another ICE instance in the ICE cluster (e.g., network switch 10 or another ICE instance in another network switch). Control plane 20 can configure ICE 16 to supply only OUT_DATA, only R_DATA, or both to packet switch 28. In some configurations (shown below), R_DATA output from ICE 16 can be fed back to ICE 16 as input.

[0044] ICE 16 may include a conversion unit 310 and a calculation unit 312. The conversion unit may be digital logic configured to convert the format of numerical data. Various formats are discussed below. The calculation unit may be digital logic configured to perform calculations on the numerical data. Digital logic may be a digital circuit defined by its inputs, its functions, and its outputs. Digital logic as used herein includes circuits (e.g., logic gates formed by transistor circuits, digital circuits formed by logic gates, etc.). Digital logic as used herein does not encompass software or abstract representations. Some embodiments of conversion unit 310 and calculation unit 312 are described below. Conversion unit 310 and calculation unit 312 may process payload data and numerical data contained in the payload data. ICE 16 may include digital logic for passing header data from input to output (“Packet Header Processing 311”). ICE 16 may receive control data supplied by control plane 20, packet switch 28, or both. Control data can configure ICE 16 for a specific ICE example (discussed below) and can specify conversion operations, calculation operations, numerical data formats, rounding modes, and group control information (e.g., group start, group end, payload offset, byte count, etc.).

[0045] Figure 3BThis is a block diagram depicting a packet 300 that can be processed by an ICE example according to some embodiments. Packet 300 may include a header 350 and a payload 352. The header 350 may include a stack or multi-level header. The payload 352 may contain numerical data 354. The numerical data 354 may be a portion of one or both of high payload data 356U and low payload data 356L. The payload 352 may have a width defined by the number of bytes (where a byte is octet). For example, some payloads may have a width of 4N bytes, where N is an integer greater than 1. Other payloads described herein may have a width of 8N bytes. The high payload data 356U and low payload data 356L may each be half the width of the payload 352. For a 4N-byte payload, the high payload data 356U and low payload data 356L may each be 2N bytes wide. For an 8N-byte payload, the high payload data 356U and the low payload data 356L can each be 4N bytes wide. The numerical data 354 can be aligned within the payload 352, for example, aligned to the most significant bit (MSB) (e.g., the bit adjacent to header 350) or the least significant bit (LSB) (e.g., the bit most opposite to the MSB). In various instances, MSB alignment can be assumed. Thus, 2N bytes of numerical data can be in the high payload data 356U of a 4N-byte payload (MSB aligned). In another instance, 4N bytes of numerical data can be in the high payload data 356U of an 8N-byte payload (MSB aligned).

[0046] Each of the high payload data 356U and the low payload data 356L may contain a set of words 3581…358 x Where x is an integer depending on the width of the payload (352 bits) and the word width. In the various examples in this paper, the word width can be 32 bits (4 bytes). Therefore, for a 4N-byte payload with a 4-byte word, x can be N / 2. For an 8N-byte payload with a 4-byte word, x can be N. Each word is 3581…358 x A number 360 can contain one or more numbers of numerical data 354. The number of digits in each word can depend on the number of bits used to represent each digit. For a 32-bit number, each word can have one digit. For a 16-bit number, each word can have two digits. For an 8-bit number, each word can have four digits. A packed integer word can be a word in which multiple integers are arranged (e.g., a 4-byte word with two 16-bit integers or four 8-bit integers). A packed floating-point word can be a word in which multiple floating-point numbers are arranged (e.g., a 4-byte word with two digits in 16-bit floating-point format or four digits in 8-bit floating-point format). A number 360 can contain multiple bytes. The order of the bytes in a number 360 can be either big-endian or little-endian.

[0047] Figure 4 This is a flowchart depicting a method 400 for processing packets in a network switch according to some embodiments. Method 400 begins at step 402, where network switch 10 receives an incoming packet. The packet's payload contains numerical data. The numbers in the numerical data may be in an input format. At step 403, packet switch 28 in network switch 10 may forward the incoming packet to a configured example (ICE example) of ICE 16. At optional step 404, the ICE example may convert the format of the numerical data from an input format to a calculated format. In some embodiments, the format conversion may be an upconversion (e.g., increasing the number of bits per digit). In other embodiments, the format conversion may be a flatconversion (e.g., converting something other than the number of bits per digit). Step 404 may be performed on all incoming packets or some incoming packets (e.g., some incoming packets may have numerical data already in a calculated format). Step 404 may be omitted, and the format of the incoming numerical data may not be converted (e.g., all incoming numerical data may already be in a calculated format).

[0048] At step 406, the ICE instance can perform a computation using selected operations. For example, an operation can be selected from a set of operations, which may include a summation operation, a summation and scaling operation, a minimum value operation, and a maximum value operation. A summation operation can be the sum of operands to produce a result. A summation and scaling operation can be the sum of operands, the result of which is multiplied by a scalar value to produce a result. A minimum value operation can determine the minimum value of the operands. A maximum value operation can determine the maximum value of the operands. In some embodiments, the ICE instance can be part of an ICE cluster that includes multiple connected ICE instances. In this case, at step 408, one or more selected operations can be used to perform the computation in step 406 across multiple ICE instances. Some embodiments of ICE clusters are described below.

[0049] At optional step 410, the ICE example may convert the format of the numerical data obtained from step 406 from the calculated format to the output format. For example, the conversion may be a down-conversion (e.g., the output format may have fewer bits per digit than the calculated format). Step 410 is optional and may be omitted (e.g., the packet may be returned to packet switch 28 for transmission without requiring format conversion of the numerical data). At step 411, the ICE example may return the packet to packet switch 28. At step 412, packet switch 28 may transmit the outgoing packet. Each payload may contain numerical data with numbers in the output format (or calculated format, if no conversion was performed at step 410).

[0050] Figure 5This is a block diagram depicting a computing unit 312 of an ICE 16 according to some embodiments. The computing unit 312 may include input interfaces 5020 and 5021, a multiplexer 504, digital logic 506, a control interface 508, and output interfaces 5180 and 5181. The control interface may be an interface for receiving and / or transmitting control data. The multiplexer may be a digital circuit that can select among its inputs to couple to its output based on control data. The input of multiplexer 504 may be coupled to input interface 5020. The output of multiplexer 504 may be coupled to the input of digital logic 506. Input interface 5021 may be coupled to another input of digital logic 506. The output of digital logic 506 may be coupled to output interface 5180. Another output of digital logic 506 may be coupled to output interface 5181. Another input of multiplexer 504 may be coupled to control interface 508. Another input of digital logic 506 may be coupled to control interface 508. Another input of the multiplexer 504 can be coupled to the output interface 5180.

[0051] Multiplexer 504 can connect its output to input interface 5020 or output interface 5180 based on control data from control interface 508. Input interface 5020 can receive numerical data from IN_DATA0 and input interface 5021 can receive numerical data from IN_DATA1. Output interface 5181 can supply numerical data to OUT_DATA. Output interface 5180 can supply numerical data to R_DATA. Therefore, the first input of digital logic 506 can receive numerical data from IN_DATA0 or R_DATA, and the second input of digital logic 506 can receive numerical data from IN_DATA1.

[0052] Digital logic 506 can receive first and second numerical data as inputs. Digital logic 506 can perform calculations on the first and second numerical data to generate third and optionally fourth numerical data. The calculations may include arithmetic operations using numbers from the first and second numerical data as operands, and the numbers generating the third and optionally fourth numerical data are the results of those operations. Digital logic 506 can supply the third and optionally fourth numerical data as outputs. Digital logic 506 can be configured via control data on control interface 508. Instance configurations may include, for example, the type of operation performed, the format used by the operands, and whether a fourth numerical data is generated. The first numerical data may be payload data from a packet of IN_DATA0 received at input A of digital logic 506. The second numerical data may be payload data from a packet of IN_DATA1 received at input B of digital logic 506. The third numerical data may be payload data from a packet of OUT_DATA supplied by output Z of digital logic 506. The fourth numerical data may be the payload data of the packets in R_DATA supplied by the output R of digital logic 506. In an embodiment, the width of the numerical data supplied at inputs A and B and by outputs Z and R may be 4N bytes.

[0053] The digital logic 506 may contain computing elements (CPEs) 5121…512 N In this example, each input A and B can receive 4N bytes of numerical data, and each output Z and R can supply 4N bytes of numerical data. Each CPE 5121…512 N It can contain two 4-byte inputs and two 4-byte outputs. (CPE 512) k It can receive input A (operand A) k The k-th 4-byte word and the input B (operand B) k The k-th 4-byte word of ) where k {1…N}. CPE 512 k Available from output Z (result Z) k The k-th 4-byte word and the result from output R (R k The k-th 4-byte word of ) where k {1…N}. Figure 5 The hollow circles in the diagram represent data splits (e.g., from 4N bytes of value to N 4B values). Figure 5 The hollow cubes in the diagram represent data concatenation (e.g., N 4B values ​​to 4N bytes). The computing element can be digital logic configured to perform arithmetic operations on numbers, as further described below. Each CPE 5121…512 NThe type of operation to be performed can be selected using control data from control interface 508.

[0054] Figure 6 This is a block diagram depicting computing elements according to some embodiments. CPE 512 k (k {1…N}) can include floating-point digital logic 6021, 6022, and 6023, integer digital logic 6024, and a multiplexer 604. CPE 512 k It can accept two operands A k and B k Operand A k and B k Each of these can be a 4-byte word. The output of the multiplexer 604 can supply the operation result Z. k and operation results R k Each of the results of the operation can be a 4-byte word. Each digital logic 6021…6024 can have a function for receiving operand A. k and B k The inputs and outputs for supplying results are provided. The output of each digital logic 6021…6024 can be coupled to a corresponding input of multiplexer 604. Multiplexer 604 can receive control data from control interface 508 to select an output from one of the digital logics 6021…6024. That is, an output from the selected digital logic is used as Z. k From CPE 512 k The output is another output from the selected digital logic as R. k From CPE 512 k Output.

[0055] The 6021 digital logic unit can perform floating-point arithmetic using numbers in 32-bit floating-point format. The floating-point format can represent both the integer and fractional parts of a number. A floating-point format may include a sign bit, an exponent of a certain number, and a fractional part (called the mantissa or significant digits). The sign bit determines whether the number is positive or negative. The exponent controls the range of values ​​(how large or small the number can be). The significant digits encode the significant bits of the number and control the precision (e.g., how many significant digits can be accurately represented). A 32-bit floating-point format may contain 32 bits for representing the number. Each 4-byte operand A k and B k It can contain a number represented in 32-bit floating-point format. Each output of the digital logic 6021 can supply a 4-byte result, which is a number represented in 32-bit floating-point format. As is well known in the art, an example 32-bit floating-point format is called FP32, which contains one sign bit, eight exponent bits, and 23 significant bits.

[0056] The Digital Logic 6022 can perform operations using numbers in 16-bit floating-point format. A 16-bit floating-point format contains 16 bits used to represent a number. Each 4-byte operand A... k and B k It can contain two numbers represented in 16-bit floating-point format. Examples of 16-bit floating-point formats include FP16 and BF16, each of which is well-known in its respective field. FP16 may contain one sign bit, five exponent bits, and 10 significant bits. BF16 may contain one sign bit, eight exponent bits, and 7 significant bits. The Digital Logic 6022 can perform two 16-bit floating-point operations in parallel (sometimes referred to as packed arithmetic).

[0057] The Digital Logic 6023 can perform operations using numbers in 8-bit floating-point format. The 8-bit floating-point format contains 8 bits used to represent a number. Each 4-byte operand A... k and B k It can contain four numbers represented in 8-bit floating-point format. Examples of 8-bit floating-point formats include various forms of FP8 known in the field, such as E5M2 and E4M3. E5M2 can contain one sign bit, five exponent bits, and two mantissa bits. E4M3 can contain one sign bit, four exponent bits, and three mantissa bits. The Digital Logic 6023 can perform four 8-bit floating-point operations in parallel.

[0058] The Digital Logic 6024 can perform operations using numbers in integer format, including 32-bit, 16-bit, and 8-bit integer formats. Each 4-byte operand A k and B k It can contain one 32-bit integer, two 16-bit integers, or four 8-bit integers. The Digital Logic 6024 can perform one 32-bit integer operation, two 16-bit integer operations in parallel, or four 8-bit integer operations in parallel.

[0059] Figure 7 This is a block diagram of digital logic 602 in a computing element according to some embodiments. Figure 6 Each of the digital logic 6021…6024 shown in the diagram may have the following characteristics: Figure 7The structure shown in the digital logic 602 is illustrated below. Digital logic 602 may include a reducer 702 and a scaler 708. The reducer 702 may include an adder 704 and a comparator 706. The number of adders 704 and comparators 706 in the reducer 702, and the number of scalers 708, may depend on the number of parallel operations performed by digital logic 602. For 32-bit operations, digital logic 602 may include one adder 704, one comparator 706, and one scaler 708. For 16-bit operations, digital logic 602 may include two adders 704, two comparators 706, and two scalers 708 operating in parallel. For 8-bit operations, digital logic 602 may include four adders 704, four comparators 706, and four scalers 708 operating in parallel.

[0060] An adder may be digital logic that computes the sum of its inputs and supplies the sum as its output. A comparator may be digital logic that compares its inputs and supplies the result of the comparison as its output (e.g., the minimum or maximum value of the inputs). A scaling element may be digital logic that computes the product of its inputs and supplies the product as its output (e.g., a multiplier). Adder 704 and comparator 706 may be part of reduction element 702, which performs reduction operations in a distributed learning system.

[0061] The input of the reduction element 702 can receive operand A k and B k (For example, a 4-byte word). Another input to the reduce element 702 can receive configuration data (shown as re_cfg) from the control interface 508. The configuration data can select whether the reduce element 702 performs addition using adder 704 or comparison using comparator 706. Adder 704 can perform comparison on A k The numbers and B k The comparator 706 performs a summation operation on the numbers in A. k The numbers and B k The numbers in the array are compared. The output of reducer 702 can be coupled to the input of scaler 708. Other inputs of scaler 708 can receive multipliers (shown as scale_cfg) from control data on control interface 508. Each scaler 708 can multiply the multiplier by the output of the corresponding adder 704. The output of scaler 708 can be coupled to the input of multiplexer 710. The output of reducer 702 (e.g., the output of adder 704 or comparator 706) can be coupled to other inputs of multiplexer 710. Another input of multiplexer 710 can receive control data (shown as mux_cfg) on ​​control interface 508. The output of multiplexer 710 can supply the result Z. kThe multiplexer 710 can be controlled to select either the output of the scaling element 708 or the output of the reducing element 702 as the result. k The output of the reducer 702 can be used as the result R. k supply.

[0062] Reduction element 702 can be configured to perform reduction operations (e.g., aggregation and reduction of intermediate computation results) in a distributed learning system. Instance reduction operations include summation, summation and scaling, minimum value operation, and maximum value operation. Reduction element 702 can be configured to perform summation using adder 704 or minimum / maximum value operation using comparator 706. The result supplied by reduction element 702 can be reduced data (e.g., summing two intermediate results to generate a sum or comparing them to generate a minimum / maximum value result). Digital logic 602 can be configured to scale the reduced data by scalar values. The output Z of digital logic 602 k It can be reduced data or reduced data as scaled. Output R k This can be reduced data. The output R from an example across a CPE from a logic 602 is... k The numerical data for R_DATA discussed above can be provided. The reduced data may omit scaling and can be used by other ICE examples in the reduction operation (if desired) before scaling is performed in the final ICE example.

[0063] For some examples of digital logic 602, adder 704 and scaling element 708 can be configured to support multiple formats of input operands. For example, for 16-bit floating-point operations, adder 704 and scaling element 708 can support both FP16 and BF16 formats. For 8-bit floating-point operations, adder 704 and scaling element 708 can support E5M2 and E4M3 formats of FP8 numbers. In some embodiments, for digital logic 602 supporting multiple input formats, adder 704 can be a Unified Multi-Format (UMF) adder, and scaling element 708 can be a UMF multiplier, as discussed further below.

[0064] Figure 8 This is a block diagram depicting a UMF adder 800 according to some embodiments. The UMF adder 800 may include digital logic 802, a data path 804, and digital logic 824. The inputs of digital logic 802 may each receive a number a, each comprising n bits. j and b j Number a j It can be the operand A discussed above. k The j-th number in the array, where j is an integer with a range depending on the number of bits (n) per digit. For n = 16 bits, j can be 1 or 2, representing 4 bytes of word A respectively. kThe first or second number in the string. For n = 8 bits, j can be 1, 2, 3, or 4, representing 4-byte word A. k The first, second, third, or fourth number in the sequence. Similarly, the number bj can be the operand B discussed above. k The j-th number in the array. The output of the digital logic 824 can supply the number rj. The number rj can be the result R discussed above. k The j-th number in the sequence.

[0065] The digital logic 802 can be configured to convert digital a j and b j The format is manipulated from the input format to the internal format. The input format can be an n-bit floating-point format (e.g., 16-bit or 8-bit). The input format can be expressed as (1, e_i, m_i), where there is a sign bit, e_i represents the number of exponent bits, and m_i represents the number of mantissa bits. The internal format can be expressed as (1, e, m), where there is a sign bit, e represents the number of exponent bits, and m represents the number of mantissa bits. The internal format can be a "super format," where e is the maximum number of exponent bits across all supported formats and m is the maximum number of mantissa bits across all supported formats. For example, the UMF adder 800 can be an n=16-bit adder supporting FP16 (1, 5, 10) and BF16 (1, 8, 7) formats. In this example, the internal format can be a (1, 8, 10) super format. In another example, the UMF adder 800 can be an n=8-bit adder supporting E5M2 (1, 5, 2) and E4M3 (1, 4, 3) FP8 formats. In this example, the internal format can be a (1, 5, 3) super format. The sign bit can be the most significant bit (MSB) of the super format. The exponent field can be aligned to the least significant bit (LSB) in the super format, where the remaining high bits are zero if e > e_i. The mantissa field can be aligned to the MSB in the super format, where the remaining low bits are zero if m > m_i. Another input of the digital logic 802 can be coupled to a control interface 508 to receive control data for selecting an input format from the supported input formats.

[0066] One output of the 802 digital logic controller can convert a j An input to data path 804 is supplied in an internal format (super format), and another output of digital logic 802 can provide b jAnother input to data path 804 is supplied in an internal format (super format). Data path 804 may be digital logic configured to perform floating-point addition of input numbers in super format. Floating-point addition may include, for example, unpacking the number into its sign, exponent, and mantissa; aligning the exponent; adding / subtracting the mantissa; normalizing the result; adjusting the exponent; rounding the result; and packing the sign, exponent, and mantissa back into the internal format. Data path 804 may include rounding modes 806 and special value handling 808. Data path 804 may support multiple rounding modes 806 selectable by control data on control interface 508. Data path 804 may support different special value handling 808 selectable by control data on control interface 508. Data path 804 may include dedicated rounding modes 806 for different supported input formats. Figure 10A Examples of different rounding modes 806 are shown in section B. Data path 804 may contain special value handling 808 for different supported input formats (e.g., non-numeric (NAN) handling, abnormal handling, overflow handling, IEEE compliance, etc.).

[0067] The output of data path 804 can supply the resulting digital value to the input of digital logic 824 in an internal format. Another input of digital logic 824 can receive control data from control interface 508. The control data can select the output format. Digital logic 824 can convert the format of its inputs to the output format selected by the control data. In this example, the output format can be the same as the input format. The output of digital logic 824 can supply digital r j .

[0068] Figure 9 This is a block diagram depicting a UMF multiplier 900 according to some embodiments. The UMF multiplier 900 may include digital logic 902, a data path 904, and digital logic 924. The inputs of digital logic 902 may each receive a number s, each comprising n bits. j and r j Numbers j This can be the j-th multiplier supplied in the control data discussed above, where j is an integer having a range depending on the number of bits (n) per digit. For n = 16 bits, j can be 1 or 2, representing the first or second digit in the 4-byte control word, respectively. For n = 8 bits, j can be 1, 2, 3, or 4, representing the first, second, third, or fourth digit in the 4-byte control word, respectively. Similarly, the number r... j The result R discussed above can be... k The j-th number in the array. The output of the digital logic 924 can supply the digital z. j Number z j This could be the result of the discussion above, Z. k The j-th number in the sequence.

[0069] The digital logic 902 can be configured to convert digital s j and r j The format is manipulated from the input format to the internal format. The input format can be an n-bit floating-point format (e.g., 16-bit or 8-bit). The input format can be expressed as (1, e_i, m_i), where there is a sign bit, e_i represents the number of exponent bits, and m_i represents the number of mantissa bits. The internal format can be expressed as (1, e, m), where there is a sign bit, e represents the number of exponent bits, and m represents the number of mantissa bits. The internal format can be a "super format," where e is the maximum number of exponent bits across all supported formats, and m is the maximum number of mantissa bits across all supported formats. For example, the UMF multiplier 900 can be an n=16-bit multiplier supporting FP16 (1, 5, 10) and BF16 (1, 8, 7) formats. In this example, the internal format can be a (1, 8, 10) super format. In another example, the UMF multiplier 900 can be an n=8-bit multiplier supporting E5M2 (1, 5, 2) and E4M3 (1, 4, 3) FP8 formats. In this example, the internal format can be a (1, 5, 3) super format. The sign bit can be the MSB of the super format. The exponent field can be LSB aligned in the super format, where the remaining high bits are zero if e > e_i. The mantissa field can be MSB aligned in the super format, where the remaining low bits are zero if m > m_i. Another input of the digital logic 902 can be coupled to the control interface 508 to receive control data for selecting the input format from the supported input formats.

[0070] One output of the digital logic 902 can convert s j An input to data path 904 is supplied in an internal format (super format), and another output in digital logic 902 can... j Another input is supplied to data path 904 in an internal format (super format). Data path 904 may be digital logic configured to perform floating-point multiplication of input numbers in super format. Floating-point multiplication may include, for example, unpacking the number into its sign, exponent, and mantissa; adding the exponents; multiplying the mantissas; normalizing the result; adjusting the exponent; rounding the result; and packing the sign, exponent, and mantissa back into the internal format. Data path 904 may include rounding modes 906 and special value handling 908. Data path 904 may support multiple rounding modes 906 selectable by control data on control interface 508. Data path 904 may support different special value handling 908 selectable by control data on control interface 508. Data path 904 may include dedicated rounding modes 906 for different supported input formats. Figure 10AExamples of different rounding modes 906 are shown in section B. Data path 904 may contain special value handling 908 for different supported input formats (e.g., NAN handling, abnormal handling, overflow handling, IEEE compliance, etc.).

[0071] The output of data path 904 can supply the resulting digital value to the input of digital logic 924 in an internal format. Another input of digital logic 924 can receive control data from control interface 508. The control data can select the output format. Digital logic 924 can convert the format of its inputs to the output format selected by the control data. In this example, the output format can be the same as the input format. The output of digital logic 924 can supply digital z j .

[0072] Figure 10A It is a table summarizing the data processing performed by a UMF adder or UMF multiplier according to some embodiments. Figure 10A This relates to a UMF adder 800 or a UMF multiplier 900 configured to support n=8-bit numbers with input formats selected from FP8_E5M2 or FP8_E4M3. For the input format FP8_E5M2, the format can be expressed as (1, 5, 2) or S.E4E3E2E1E0.M1M2, representing the number of sign (S), exponent (E), and mantissa (M) bits. For the input format FP8_E4M3, the format can be expressed as (1, 4, 3) or S.E3E2E1E0.M1M2M3. For FP8_E5M2, the internal superformat can be (1, 5, 3) or S.E4E3E2E1E0.M1M20. For FP8_E4M3, the internal superformat can be (1, 5, 3) or S.E4E3E2E1E0.M1M2M3. The exponent path in the data path can handle 5 exponents, and the significant path in the data path can handle 4 significant digits (mantissa). For each of FP8_E5M2 and FP8_E4M3, the rounding format can be selected as {4 significant digits, 1 rounding bit, 1 sticky bit}. For FP8_E5M2, the rounding mode can be selected to round to 2 mantissas. For FP8_E4M3, the rounding mode can be selected to round to 3 mantissas.

[0073] Figure 10B It is a table summarizing the data processing performed by a UMF adder or UMF multiplier according to some embodiments. Figure 10BThis relates to a UMF adder 800 or a UMF multiplier 900 configured to support n=16-bit numbers with input formats selected from BF16 or FP16. For the BF16 input format, the format can be expressed as (1, 8, 7) or S.E7E6E5E4E3E2E1E0.M1M2M3M4M5M6M7. For the FP16 input format, the format can be expressed as (1, 5, 10) or S.E4E3E2E1E0.M1M2M3M4M5M6M7M8M9M 10 For BF16, the internal super format can be (1, 8, 10) or S.E7E6E5E4E3E2E1E0.M1M2M3M4M5M6M7000. For FP16, the internal super format can be (1, 8, 10) or S.000E4E3E2E1E0.M1M2M3M4M5M6M7M8M9M. 10 The exponent path in the data path can handle an 8-bit exponent, and the significant path in the data path can handle 11 significant digits (mantissa). For each of BF16 and FP16, the rounding format can be selected as {11 significant digits, 1 rounding bit, 1 sticky bit}. For BF16, the rounding mode can be selected to round to 7 mantissas. For FP16, the rounding mode can be selected to round to 10 mantissas.

[0074] Figure 11 This is a block diagram depicting a conversion unit 310 according to some embodiments. The conversion unit 310 may include digital logic 1102, digital logic 1105, and digital logic 1106. Digital logic 1102 may receive inputs of receiving numerical data (e.g., 4N bytes of numerical data). Digital logic 1106 may have outputs of supplying numerical data (e.g., up to 8N bytes of numerical data, such as 2N bytes, 4N bytes, or 8N bytes of numerical data). The input numerical data may be in an input format, and the output numerical data may be in an output format. The output format may contain more bits per digit than the input format (up-conversion), fewer bits per digit than the input format (down-conversion), or the same number of bits per digit as the input format (flat conversion). The output of digital logic 1102 may be coupled to the input of digital logic 1105. The output of digital logic 1105 may be coupled to the input of digital logic 1106. The input of digital logic 1102 may receive control data from a control interface 508. The input of digital logic 1105 may receive control data from the control interface 508. The inputs of digital logic 1106 can receive control data from control interface 508.

[0075] Digital logic 1105 may contain digital logic 11040…11043. Each digital logic 1104… kIt may contain a set of N conversion elements (CVEs). The conversion elements may be digital logic that can accept numbers in an input format and output numbers in an output format different from the input format. Each CVE 11040 may contain 4 bytes of input (e.g., an 8-bit, 16-bit, or 32-bit operand) and 4 bytes of output (e.g., an 8-bit, 16-bit, or 32-bit result). Each CVE 11041 may contain 2 bytes of input (e.g., an 8-bit or 16-bit operand) and 4 bytes of output (e.g., an 8-bit, 16-bit, or 32-bit result). Each CVE 11042 and CVE 11043 may contain 1 byte of input (e.g., an 8-bit operand) and 2 bytes of output (e.g., an 8-bit or 16-bit result). Each CVE may be implemented using a UMF converter, as described in the embodiments below. Digital logic 1102 can output four sets of inputs to digital logic 1105, such as 4N bytes of input to CVE 11040, 2N bytes of input to CVE 11041, N bytes of input to CVE 11042, and N bytes of input to CVE 11043. Digital logic 1106 can receive 4N bytes of output from CVE 11040, 4N bytes of output from CVE 11041, 2N bytes of output from CVE 11042, and 2N bytes of output from CVE 11043. Digital logic 1102 can perform multiplexing operations, and digital logic 1106 can perform packetization operations, such as... Figure 12 The table summarizes the following. Multiplexing and packetization operations can be controlled by the corresponding control data on control interface 508. The configuration of CVEs in digital logic 1105 (e.g., UMF CVE) can be controlled by the control data on control interface 508.

[0076] Selectable input data formats include 32-bit, 16-bit, and 8-bit floating-point formats (e.g., FP32, BF16, FP16, FP8(E5M2, E4M3)), and 32-bit, 16-bit, and 8-bit signed and unsigned integer formats. Upconversion may involve converting the input format to another format twice its width (e.g., 16-bit to 32-bit, 8-bit to 16-bit). Downconversion may involve converting the input format to another format half its width (e.g., 32-bit to 16-bit, 16-bit to 8-bit). Flatconversion may involve converting the input format to another format of the same width (e.g., FP8_E5M2 to FP8_E4M3 or vice versa, BF16 to FP16 or vice versa).

[0077] Figure 12This is a table summarizing the operation of the conversion unit 310 according to some embodiments. The input can be 4N bytes of numerical data with 32-bit format numbers, 4N bytes of numerical data with 16-bit format numbers, or 4N bytes of numerical data with 8-bit format numbers. These formats are discussed above. Digital logic 1102 can provide a multiplexing stage in which: (1) for 32-bit, 4N-byte data, all 4N-byte inputs are selected as inputs to CVE 11040; (2) for 16-bit, 4N-byte data, the lower 2N bytes of the input are selected as inputs to CVE 11040 and the higher 2N bytes of the input are selected as inputs to CVE 11041; and (3) for 8-bit, 4N-byte inputs, each N-byte block is used as input to four sets of CVE 11040…11043.

[0078] Digital logic 1105 provides a conversion stage. For 32-bit, 4N-byte input data, digital logic 1105 can perform a flat conversion or a downconversion to 16 bits. For a flat conversion, each CVE 11040 can have 32-bit input data, which is converted to 32-bit result data. For a downconversion, each CVE 11040 can have 32-bit input data, which is converted to 16-bit result data. Digital logic 1106 provides a packing stage. For a flat conversion, digital logic 1106 can select each 32-bit result from CVE 11040 and generate a 4N-byte output (with 32-bit numbers). For a downconversion, digital logic 1106 can select each 16-bit result from CVE 11040 and generate a 2N-byte output (with 16-bit numbers).

[0079] For 16-bit, 4N-byte input data, digital logic 1105 can perform a 32-bit upconversion, a flat conversion, or an 8-bit downconversion. For upconversion, each CVE 11040 and 11041 has 16-bit input data, which is converted to 32-bit result data. For flat conversion, each CVE 11040 and 11041 can have 16-bit input data, which is converted to 16-bit result data. For downconversion, each CVE 11040 and 11041 can have 16-bit input data, which is converted to 8-bit result data. For upconversion, digital logic 1106 can select each 32-bit result from CVE 11040 and 11041 and generate an 8N-byte output (with 32 bits). For flat conversion, digital logic 1106 can select each 16-bit result from CVEs 11040 and 11041 and generate 4N bytes of output (with 16-bit numbers). For down conversion, digital logic 1106 can select each 8-bit result from CVEs 11040 and 11041 and generate 2N bytes of output (with 8-bit numbers).

[0080] For 8-bit, 4N-byte input data, digital logic 1105 can perform an upconversion to 16 bits or a flat conversion to 8 bits. For the upconversion, each CVE 11040…11043 has 8-bit input data, which is converted to 16-bit result data. For the flat conversion, each CVE 11040…11043 has 8-bit input data, which is converted to 8-bit result data. For the upconversion, digital logic 1106 can select each 16-bit result from CVE 11040…11043 and generate an 8N-byte output (with 16-bit numbers). For the flat conversion, digital logic 1106 can select each 8-bit result from CVE 11040…11043 and generate a 4N-byte output (with 8-bit numbers).

[0081] Figure 13 This is a block diagram depicting a UMF converter 1300 according to some embodiments. The UMF converter 1300 may include digital logic 1302 and digital logic 1308. The input of digital logic 1302 may receive numerical data to be converted. The output of digital logic 1308 may supply numerical data having an output format. Digital logic 1302 may convert the input format from an input format to an intermediate format (IF). The intermediate format may be a super format covering the range and precision of all input formats. The output of digital logic 1302 may supply numerical data with IF to the input of digital logic 1308. Digital logic 1308 may convert the numerical data with IF to an output format. Another input of digital logic 1302 may receive control data from a control interface 508. The control data may select the input format of the input data. Another input of digital logic 1308 may receive control data from the control interface 508. The control data may select the rounding mode and output format.

[0082] Digital logic 1302 converts an input format (in_format) to IF floating-point format (IFP). Digital logic 1302 may contain digital logics 13041…13046. Digital logic 13041 converts a number in FP32 format to IFP. Digital logic 13042 converts a number in BF16 format to IFP. Digital logic 13043 converts a number in FP16 format to IFP. Digital logic 13044 converts a number in FP8_E5M2 format to IFP. Digital logic 13045 converts a number in FP8_E4M3 format to IFP. Digital logic 1302 may contain digital logic 1306 configured to use sign extension to convert 32, 16, or 8-bit integers to extended-bit integers. Sign extension increases the number of bits in a binary number while preserving the sign of the number. Digital logic 13046 converts a number in INText format to IFP.

[0083] Digital logic 1308 converts an IFP to an output format. Digital logic 1308 may include digital logic 13101…13106. Digital logic 13101 converts an IFP-formatted number to FP32. Digital logic 13102 converts an IFP-formatted number to BF16. Digital logic 13103 converts an IFP-formatted number to FP16. Digital logic 13104 converts an IFP-formatted number to FP8_E5M2. Digital logic 13105 converts an IFP-formatted number to FP8_E4M3. Digital logic 13106 converts an IFP-formatted number to INText. Digital logic 1308 may include digital logic 1312 configured to convert INText to INT (e.g., a 32, 16, or 8-bit integer).

[0084] Figure 14 This is a table summarizing the format conversions of CVEs based on some embodiments. For CVE 11040 (CVE-0), the input format conversions can be: FP32 to IFP; BF16 to IFP; FP16 to IFP; E5M2 to IFP; E4M3 to IFP; and INT33 to IFP. For CVE 11040 (CVE-0), the intermediate format can be FP40 (1, 8, 31). For CVE 11040 (CVE-0), the output format conversions can be: IFP to FP32; IFP to BF16; IFP to FP16; IFP to E5M2; IFP to E4M3; and IFP to INT33.

[0085] For CVE 11041 (CVE-1), the input format conversion can be: BF16 to IFP; FP16 to IFP; E5M2 to IFP; E4M3 to IFP; and INT33 to IFP. For CVE 11041 (CVE-1), the intermediate format can be FP40 (1, 8, 31). For CVE 11041 (CVE-1), the output format conversion can be: IFP to FP32; IFP to BF16; IFP to FP16; IFP to E5M2; IFP to E4M3; and IFP to INT33.

[0086] For CVE 11042 (CVE-2), the input format conversion can be: E5M2 to IFP; E4M3 to IFP; and INT17 to IFP. For CVE 11042 (CVE-2), the intermediate format can be FP24 (1, 8, 15). For CVE 11042 (CVE-2), the output format conversion can be: IFP to BF16; IFP to FP16; IFP to E5M2; IFP to E4M3; and IFP to INT17.

[0087] For CVE 11043 (CVE-3), the input format conversion can be: E5M2 to IFP; E4M3 to IFP; and INT17 to IFP. For CVE 11043 (CVE-3), the intermediate format can be FP24 (1, 8, 15). For CVE 11043 (CVE-3), the output format conversion can be: IFP to BF16; IFP to FP16; IFP to E5M2; IFP to E4M3; and IFP to INT17.

[0088] Figure 15A This is a block diagram depicting an ICE example 1500A according to some embodiments. The ICE example may be a configured ICE 16. ICE example 1500A may include a conversion unit 310 configured to upconvert IN_DATA1. ICE example 1500A may include two parallel computing units 3120 and 3121. Open circuits indicate data splitting (e.g., one 8N-byte value to two 4N-byte values) and hollow squares indicate data concatenation (e.g., two 4N-byte values ​​to one 8N-byte value). IN_DATA0 may be 8N-byte numerical data and IN_DATA1 may be 4N-byte numerical data. Computation unit 3120 may receive the lower 4N bytes from IN_DATA0 and computation unit 3121 may receive the higher 4N bytes from IN_DATA0. Conversion unit 310 may convert the format of IN_DATA1 to twice the width, resulting in an output of 8N-byte data. In some embodiments, ICE example 1500A may include another conversion unit 310 configured to upconvert IN_DATA0. IN_DATA0 can be 4N bytes of numerical data. Another conversion unit 310 can convert the format of IN_DATA0 to twice the width, resulting in an output of 8N bytes of data. Calculation unit 3120 can receive the lower 4N bytes from conversion unit 310, and calculation unit 3121 can receive the higher 4N bytes from conversion unit 310. OUT_DATA can be 8N bytes of data and can be a concatenation of 4N bytes of data output from calculation unit 3120 and 4N bytes of data output from calculation unit 3121. Similarly, R_DATA can be 8N bytes of data and can be a concatenation of 4N bytes of reduced data from calculation unit 3120 and 4N bytes of reduced data from calculation unit 3121. Control interface 508 is omitted for clarity but is described in detail above.

[0089] Figure 15BThis is a block diagram depicting an ICE example 1500B according to some embodiments. ICE example 1500B may include a calculation unit 312 and a conversion unit 310 configured for downconversion. The inputs of the calculation unit 312 may receive IN_DATA0 and IN_DATA1, each a 4N-byte numerical data. The reduced data output of the calculation unit 312 may supply R_DATA as 4N-byte data. Another output of the calculation unit 312 may be coupled to the input of the conversion unit 310 (supplying 4N-byte data). The conversion unit 310 may downconvert the format of the 4N-byte numerical data and output 4N-byte OUT_DATA.

[0090] Figure 15C This is a block diagram depicting an ICE example 1500C according to some embodiments. ICE example 1500C may include a conversion unit 310 configured to upconvert IN_DATA1. ICE example 1500C may include parallel computing units 3120 and 3121. ICE example 1500C may include parallel conversion units 3101 and 3102 configured to perform downconversion. IN_DATA0 may be 8N bytes of numerical data and IN_DATA1 may be 4N bytes of numerical data. Computation unit 3120 may receive the lower 4N bytes from IN_DATA0 and computation unit 3121 may receive the higher 4N bytes from IN_DATA0. Conversion unit 310 may convert the format of IN_DATA1 to twice its width, resulting in an output of 8N bytes of data. In some embodiments, ICE example 1500C may include another conversion unit 310 configured to upconvert IN_DATA0. IN_DATA0 may be 4N bytes of numerical data. Another conversion unit 310 can convert the IN_DATA0 format to twice the width, resulting in an output of 8N bytes of data. Calculation unit 3120 can receive the lower 4N bytes from conversion unit 310, and calculation unit 3121 can receive the higher 4N bytes from conversion unit 310. The reduced data output (4N bytes) from each of calculation units 3120 and 3121 can be concatenated and supplied as R_DATA (8N bytes). Another data output from calculation unit 3120 can be coupled to the input (4N bytes of data) of conversion unit 3101. Another data output from calculation unit 3121 can be coupled to the input (4N bytes of data) of conversion unit 3102. Down-conversion by conversion units 3101 and 3102 can reduce the width from 4N bytes to 2N bytes. The outputs of conversion units 3101 and 3102 can be concatenated and supplied as OUT_DATA (4N bytes).

[0091] Return to reference Figure 3ATogether with the ICE cluster, the ICE 16 can support port groups within a single session. A session can supply input numerical data via one or more ports in the group. The ports in the group can operate at different speeds. In some embodiments, the ICE 16 can support multiple sessions, each with a different port group. The control plane 20 can associate different port groups with the ICE 16 to process input numerical data across different sessions. In some embodiments, the ICE 16 can support multiple sessions with the same port group. One session can supply one set of input data, and another session can supply another set of input data.

[0092] In some embodiments, multiple ICE 16s may be configured as ICE instances and coupled as an ICE cluster (or super ICE instance). The ICE cluster may be a serial chain cluster, a parallel tree cluster, a multi-port shared cluster, a fully connected shared cluster, or a cluster with local data storage, as described below.

[0093] Figure 16A This is a block diagram depicting an ICE cluster 1600A according to some embodiments. The ICE cluster 1600A is an instance of a serial chain cluster. In an instance, the ICE cluster 1600A may contain four ICE instances 16020…16023. For example, each ICE instance 16020…16023 may be as follows: Figure 15C The example shown is ICE example 1500C. ICE example 16020 can receive a first input data (IN_0) at an upconversion input and optionally a second input data (IN_RED) at a non-conversion input. The second input data (IN_RED) can be reduced data from another ICE example (if present). The reduced output from ICE example 16020 can be coupled to the non-conversion input (R_DATA0) of ICE example 16021. The conversion input of ICE example 16021 can receive a third input data (IN_1). The reduced output from ICE example 16021 can be coupled to the non-conversion input of ICE example 16022. The conversion input of ICE example 16022 can receive a fourth input data (IN_2). The reduced output from ICE example 16022 can be coupled to the non-conversion input of ICE example 16023. The conversion input of ICE example 16023 can receive a fifth input data (IN_3). The conversion output of the ICE 16023 example can be supplied to OUT_DATA. Any number of ICE examples can be formed into a serial chain cluster.

[0094] Figure 16BThis is a block diagram depicting an ICE cluster 1600B according to some embodiments. The ICE cluster 1600B is an instance of a parallel tree cluster. In an instance, the ICE cluster 1600B may contain ICE examples 16040…16042. For example, each ICE example 16040…16041 may be as follows: Figure 15A The ICE example 1500A and ICE example 16042 shown in the document can be as follows: Figure 15B The example shown is ICE example 1500B. ICE example 16040's inputs can receive data IN_0 and IN_1. ICE example 16041's inputs can receive data IN_2 and IN_3. The reduced data output of ICE 16040 can supply R_DATA0 to the input of ICE 16042. The reduced data output of ICE 16041 can supply R_DATA1 to another input of ICE 16042. The inputs of ICE 16042 can be non-converted inputs. The converted output of ICE 16042 can supply OUT_DATA.

[0095] Figure 16C This is a block diagram depicting an ICE cluster 1600C according to some embodiments. The ICE cluster 1600C is an example of a multi-port shared cluster. The ICE cluster 1600C may include a scheduler 1606 and ICE examples 16080…16083. For example, each ICE example 16080…16083 may be as follows: Figure 15CThe ICE example 1500C is shown in the diagram. The scheduler can be configured to schedule digital logic inputs to be connected to outputs. The hollow ladder diagram of scheduler 1606 represents a multiplexer. Scheduler 1606 can receive inputs IN_0, IN_1, IN_2, and IN_3. Inputs IN_0…IN_3 can be coupled to the corresponding inputs of ICE examples 16080…16083. Scheduler 1606 can receive input IN_RED (e.g., a reduced input from another ICE example or cluster). Input IN_RED can be scheduled across other inputs of ICE examples 16080…16083. When input IN_RED is not connected to a given ICE example, the given ICE example can receive reduced data from a previous ICE example in the serial chain. In some configurations, control plane 20 can associate high-speed (bandwidth) ports of packet switch 28 with multiple interfaces of the ICE cluster. In the example of ICE cluster 1600C, high-speed ports can be associated with inputs of ICE examples 16080…16083. Grouped data can be evenly distributed across inputs IN_0, IN_1, IN_2, and IN_3. Input IN_RED can be a high-speed reduced input from another ICE example or cluster. Input IN_RED can be evenly distributed and scheduled to other inputs of ICE examples 16080…16083. The ICE cluster 1600C can be expanded into a fully connected ICE cluster, where the scheduler 1606 can connect each of the multiple inputs to any of the multiple outputs.

[0096] Figure 16D This is a block diagram depicting an ICE cluster 1600D according to some embodiments. ICE cluster 1600D is an instance of an ICE cluster with local data storage. ICE cluster 1600D may include ICE examples 16220…16222. For example, each ICE example 16220 and 16221 may include, Figure 15A The ICE example 1500A shown in the document, and ICE example 16222 can be as follows: Figure 15BThe example shown is ICE example 1500B. Each ICE example 16220 and 16221 can be coupled to a local data memory. One input of ICE example 16220 can receive input data IN_0. Another input of ICE example 16220 can be coupled to the output of data memory (DM) 16200. One input of ICE example 16221 can receive input data IN_1. Another input of ICE example 16221 can be coupled to the output of data memory 16201. The reduced data outputs of ICE examples 16220 and 16221 can be coupled to the input of ICE example 16222. Another output of ICE example 16220 can be coupled to the input of data memory 16200. Another output of ICE example 16221 can be coupled to the input of data memory 16201. The output of ICE 16222 can supply OUT_DATA. Data memories 16200 and 16201 can store temporary reduced results from the outputs of their respective ICE examples over time. Different results can be applied to the inputs of the corresponding ICE examples over time. Local data storage can be used to store temporary reduced results per port group or per session. In some configurations, control plane 20 can associate port groups from packet switch 28 with one input of an ICE example. In an instance of ICE cluster 1600D, packet data from port groups can be sequentially scheduled (by the scheduler) to input data IN_0. Corresponding temporary reduced results for the same session can be scheduled (by the scheduler) from data storage 16200 to another input of ICE example 16220. The reduced result output of ICE example 16220 can be stored back in data storage 16200 or output to the next ICE example 16222. Similarly, packet data from another port group can be sequentially scheduled to input data IN_1; and corresponding temporary reduced results for the same session can be scheduled from data storage 16201 to another input of ICE example 16221. The reduced output of ICE example 16221 can be stored back in data storage 16201 or output to the next ICE example 16222. ICE example 16222 can aggregate and reduce the results from the same session from ICE examples 16220 and ICE example 16221. ICE examples with local storage can be combined into more complex ICE clusters, such as star clusters or daisy-chain clusters.

[0097] While some processes and methods having various operations have been described, one or more embodiments also relate to apparatus or devices for performing these operations. The apparatus may be specifically constructed for the desired purpose, or the apparatus may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Various general-purpose machines may be used with computer programs written according to the teachings herein, or it may be more convenient to construct more specialized apparatus to perform the desired operations.

[0098] As used herein, the phrase “at least one of…” preceding a series of items (where the terms “and” or “or” are used to separate any of the items) modifies the entire list, not each member of the list (i.e., each item). The phrase “at least one of…” does not require selection of at least one of every listed item; rather, the phrase allows for the inclusion of at least one of any of the items and / or at least one of any combination of the items. For example, the phrases “at least one of A, B, and C” or “at least one of A, B, or C” each refer to only A, only B, or only C; and / or any combination of A, B, and C. In examples where the desired selection is “at least one of each of A, B, and C” or alternatively “at least one of A, at least one of B, and at least one of C,” this is explicitly stated.

[0099] As used herein, the term “coupling” and its derivatives include: (a) electrical and communication coupling; and (b) does not imply a direct connection but may include intervening elements unless described as “direct coupling”.

[0100] Although one or more embodiments of the invention have been described in some detail for ease of understanding, certain changes may be made within the scope of the claims. Therefore, the described embodiments should be considered illustrative rather than restrictive, and the scope of the claims is not limited to the details given herein, but may be modified within the scope of the claims and their equivalents. In the claims, elements and / or steps do not imply any particular order of operations unless expressly stated in the claims.

[0101] The boundaries between components, operations, and data storage are somewhat arbitrary, and specific operations are described within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of this invention. Generally, structures and functionalities presented as individual components in the exemplary configuration can be implemented as combined structures or components. Similarly, structures and functionalities presented as individual components can be implemented as individual components. These and other variations, additions, and modifications fall within the scope of the appended claims.

Claims

1. A data plane for a network switch, comprising: A packet switch configured to process packets and including a first port and a second port; A first processor, coupled to the control plane of the packet switch and the network switch, the first processor comprising: A first digital circuit is configured to: receive a first payload of a first packet imported into the first port; and convert first numerical data in the first payload from a first format to a second format; and A second digital circuit is configured to receive first numerical data and second numerical data from the first digital circuit; and to perform a first calculation operation using the first and second numerical data as input. The first processor is configured to return a second packet to the packet switch for export from the second port, wherein the second payload of the second packet contains at least part of the numerical data generated by the first computation operation.

2. The data plane of claim 1, wherein the second format and the first computation operation are selected by the configuration of the first processor by the control plane.

3. The data plane according to claim 1, wherein: The second digital circuit is configured to perform the first calculation operation to generate third numerical data having the second format; The first processor further includes a third digital circuit configured to: receive the third numerical data from the second digital circuit; and convert the third numerical data from the second format to a third format; and The numerical data in the second payload is the third numerical data output from the third digital circuit.

4. The data plane according to claim 1, wherein the first processor further comprises: A third digital circuit is configured to: receive a third payload of a third packet imported into a third port of the packet switch, the third payload containing the second numerical data; and convert the second numerical data in the third payload from a third format to the second format. The second digital circuit is configured to receive the second numerical data from the third digital circuit.

5. The data plane of claim 1, wherein the first processor is configured to perform the first calculation operation after performing the second calculation operation, and wherein the third numerical data is the result of the second calculation operation.

6. The data plane of claim 1, wherein the second digital circuitry is configured to perform the first computational operation to generate the third numerical data, and wherein the data plane further comprises: A second processor, coupled to the packet switch and the control plane, is configured to: receive a third payload of the third packet imported from the third port, the third payload containing fourth numerical data; and perform the second calculation operation using the third and fourth numerical data as input.

7. The data plane of claim 1, wherein the second digital circuitry is configured to perform the first computational operation to generate the third numerical data, and wherein the data plane further comprises: A second processor, coupled to the packet switch and the control plane, is configured to: receive a third payload of the third packet imported from the third port, the third payload containing the fourth numerical data; receive a fourth payload of the fourth packet imported from the fourth port, the fourth payload containing fifth numerical data; and perform a second calculation operation with the fourth and fifth numerical data as input to generate sixth numerical data. and A third processor, coupled to the packet switch and the control plane, is configured to receive the third and sixth numerical data and to perform a third calculation operation using the third and sixth numerical data as input.

8. The data plane of claim 1, wherein the packet switch comprises a plurality of ports, the plurality of ports including the first and second ports, and wherein the data plane further comprises: A plurality of processors, including the first processor; and A scheduler, which is coupled between the plurality of ports and the plurality of processors.

9. The data plane of claim 1, wherein the second digital circuitry is configured to perform the first computational operation to generate the third numerical data, and wherein the data plane includes a memory configured to store the third numerical data.

10. The data plane according to claim 1, wherein the first format includes a first digit and a first numerical representation, and wherein the second format includes a second digit and a second numerical representation.

11. A network switch, comprising: The control plane, which includes the central processing unit (CPU) and memory; and A data plane comprising a packet switch and a first processor, the packet switch being configured to process packets and including a first port and a second port, the first processor being coupled to the packet switch and the control plane, the first processor comprising: A first digital circuit is configured to: receive a first payload of a first packet imported into the first port; and convert first numerical data in the first payload from a first format to a second format. and A second digital circuit is configured to: receive first numerical data and second numerical data from the first digital circuit; and perform a first calculation operation using the first and second numerical data as input. The first processor is configured to return the second packet to the packet switch for export from the second port, and the second payload of the second packet contains at least part of the numerical data generated by the first computation operation; and The second format and the first computation operation are selected by the configuration of the first processor by the control plane.

12. The network switch according to claim 11, wherein: The second digital circuit is configured to perform the first calculation operation to generate third numerical data having the second format; The first processor further includes a third digital circuit configured to: receive the third numerical data from the second digital circuit; and convert the third numerical data from the second format to a third format; and The numerical data in the second payload is the third numerical data output from the third digital circuit.

13. The network switch of claim 11, wherein the first processor further comprises: A third digital circuit is configured to: receive a third payload of a third packet imported into a third port of the packet switch, the third payload containing the second numerical data; and convert the second numerical data in the third payload from a third format to the second format. The second digital circuit is configured to receive the second numerical data from the third digital circuit.

14. The network switch of claim 11, wherein the first processor is configured to perform the first calculation operation after performing the second calculation operation, and wherein the third numerical data is the result of the second calculation operation.

15. The network switch of claim 11, wherein the second digital circuitry is configured to perform the first computational operation to generate the third numerical data, and wherein the data plane further comprises: A second processor, coupled to the packet switch and the control plane, is configured to: receive a third payload of the third packet imported from the third port, the third payload containing fourth numerical data; and perform the second calculation operation using the third and fourth numerical data as input.

16. The network switch of claim 11, wherein the second digital circuitry is configured to perform the first computational operation to generate the third numerical data, and wherein the data plane further comprises: A second processor, coupled to the packet switch and the control plane, is configured to: receive a third payload of the third packet imported from the third port, the third payload containing fourth numerical data; receive a fourth payload of the fourth packet imported from the fourth port, the fourth payload containing fifth numerical data; and perform a second calculation operation with the fourth and fifth numerical data as input to generate sixth numerical data. and A third processor, coupled to the packet switch and the control plane, is configured to receive the third and sixth numerical data and to perform a third calculation operation using the third and sixth numerical data as input.

17. The network switch of claim 11, wherein the packet switch comprises a plurality of ports, the plurality of ports including the first and second ports, and wherein the data plane further comprises: A plurality of processors, including the first processor; and A scheduler, which is coupled between the plurality of ports and the plurality of processors.

18. The network switch of claim 11, wherein the second digital circuitry is configured to perform the first computational operation to generate the third numerical data, and wherein the data plane includes a memory configured to store the third numerical data.

19. A method for processing packets in a network switch, the method comprising: Processing packets in the packet switch of the network switch; and A first processor coupled to the control plane of the packet switch and the network switch controls the processing of the packet switch in the following manner: At the first digital circuit of the first processor, the first payload of the first packet introduced into the first port of the packet switch is received; The first digital circuit converts the first numerical data in the first payload from a first format to a second format. At the second digital circuit, the first numerical data and the second numerical data from the first digital circuit are received; The second digital circuit performs a first calculation operation using the first and second numerical data as input. and The second packet is returned from the first processor to the packet switch for export from the second port of the packet switch, and the second payload of the second packet contains at least part of the numerical data generated by the first computation operation.

20. The method of claim 19, further comprising: The control plane configures the first processor to select the second format and the first computation operation.

Images