Sparse SIMD Cross-Lane Processing Unit

Abstract

Aspects of the disclosure are directed to a cross-lane processing unit (XPU) for performing data-dependent operations across multiple data processing lanes of a processor. Rather than executing operation-specific circuitry for each data-dependent operation, the XPU can be configured to perform different operations in response to input signals that constitute individual operations performed by processing cells and crossbars arranged as a stacked network in the XPU. Each processing cell can receive and process data across multiple data processing lanes. Aspects of the disclosure include configuring the XPU to use a vector sort network to perform duplicate counting, eliminating the need to configure separate XPUs for sorting and duplicate counting.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 17 / 972,663, filed October 25, 2022, which claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 279,262, filed November 15, 2021, the disclosure of which is incorporated herein by reference. [Background technology]

[0002] background A single instruction multiple data (SIMD) processing unit is a type of processing unit for parallel processing of multiple data inputs by performing the same operation on each of the inputs. The operations accelerated by a SIMD processing unit are predetermined at the time of designing the SIMD processing unit. A SIMD processing unit can involve sparse computations that can frequently use the following computational primitives: duplicate count / histogram; sorting; peculiarization; scan / segmented scan; and / or partitioning. To obtain good performance in sparse computations, these primitives should be parallelized on a SIMD architecture in a composable fashion. The current state of the art cannot provide such parallelization. Summary of the Invention [Problem to be solved by the invention]

[0003] Quick Overview The disclosed aspects are directed to a cross-lane processing unit (XPU) for performing single instruction multiple data (SIMD) data-dependent operations across multiple data processing lanes of a processor. Rather than physically fabricating operation-specific circuits for each data-dependent operation, the XPU can be configured to perform different operations in response to an input signal by configuring processing cells to perform individual operations and arranging the crossbar as a stacked network in the XPU. The disclosed aspects provide for identifying unique values ​​in an input vector while also performing duplicate counting of unique and non-unique values ​​in the vector by reusing the same stacked network configuration in the XPU. Each processing cell can receive and process data across multiple data processing lanes. The XPU described herein eliminates the need for hardware to be separately configured to sort, e.g., identify unique values ​​in an input vector and count duplicate values ​​in the input vector. The XPU can be executed as part of a hardware circuit and complements computation of dense data structures such as dense matrices with accelerated processing of sparse data structures such as sparse vectors or matrices.

[0004] An aspect of the disclosure provides a hardware circuit. The hardware circuit includes a plurality of stages, each stage including a crossbar and two or more cells. The hardware circuit further includes a plurality of data processing lanes for streaming respective data from an upstream input to a downstream destination via the plurality of cells of the plurality of stages and the plurality of crossbars. The hardware circuit is configured to receive input data from the upstream input along the plurality of data processing lanes and to receive a first instruction for performing a first operation. In response to receiving the first instruction, for each stage, the hardware circuit is further configured to transmit a respective second instruction to a respective processing cell of that stage, each cell being configured to perform a respective second operation in response to receiving an input from the respective data processing lane, and the hardware circuit is further configured to transmit a respective third instruction to a respective crossbar for that stage, the crossbar being configured to permute an output from each cell of that stage to a cell of a next stage along the plurality of data processing lanes. The hardware circuitry is also configured to perform the first operation by processing the received input data along the multiple data processing lanes and the multiple cells configured to perform respective second operations.

[0005] In one example, each cell is configured to receive a respective first input operand from a respective data processing lane passing through the cell and a respective second input operand from a respective crossbar of an upstream stage of the cell. In another example, a downstream destination of data for the multiple data processing lanes is a vector processing unit, the vector processing unit configured to perform a single instruction multiple data vector operation on output data of the hardware circuit.

[0006] In yet another example, each of the cells is configured to perform one or more of a plurality of predetermined primitive operations in response to the one or more received instructions. The hardware circuit further includes a plurality of control cells. In transmitting the respective second instructions to the respective processing cells, the hardware circuit is configured to generate and transmit, by each control cell, a respective control signal to each processing cell based on a first operation specified by the first instruction. In yet another example, in generating and transmitting the respective control signal by each control cell, the hardware circuit is configured to generate a respective control signal for each processing cell to perform one of a respective arithmetic operation, a comparison operation, and a bypass operation based on at least one of a stage in which the processing cell resides or a data processing lane passing through the processing cell. In yet another example, the plurality of cells and the plurality of crossbars form a processing network of connected cells across the plurality of stages and the plurality of data processing lanes, the processing network of connected cells being configured to receive input data and generate respective output data in accordance with performing a first operation on the input data. In yet another example, the processing network of connected cells is configured to perform a combined vector sort and duplicate count operation. The combined operations include the steps of a processing network receiving an input vector of elements, and the processing network generating as output data specifying a sorted output vector and a count of duplicate elements in the input vector.

[0007] In yet another example, the input data includes sparse vector data, and after transmitting the respective second instruction and the respective third instruction, the hardware circuit is configured to perform one of a vector scan, a vector sum, a vector sort, or a vector duplicate count.

[0008] Another aspect of the disclosure provides a system including a hardware circuit, the hardware circuit including a plurality of stages, each stage including a crossbar and two or more cells, the hardware circuit including a plurality of data processing lanes for streaming respective data from an upstream input to a downstream destination through the plurality of cells of the plurality of stages and the plurality of crossbars. The hardware circuit is configured to receive input data from an upstream input along the plurality of data processing lanes and to receive a first instruction for performing a first operation. The hardware circuit is further configured to transmit, for each stage, a respective second instruction to a respective processing cell of that stage in response to receiving the first instruction, each cell configured to perform a respective second operation in response to receiving an input from a respective data processing lane and to transmit a respective third instruction to a respective crossbar for that stage, the crossbar configured to permute an output from each cell of that stage to a cell of a next stage along the plurality of data processing lanes. The hardware circuit is also configured to perform the first operation by processing the received input data along the plurality of data processing lanes and the plurality of cells configured to perform the respective second operations.

[0009] In one example, each cell is configured to receive a respective first input operand from a respective data processing lane passing through the cell and a respective second input operand from a respective crossbar of an upstream stage of the cell. In another example, a downstream destination of data for the multiple data processing lanes is a vector processing unit, the vector processing unit configured to perform a single instruction multiple data vector operation on output data of the hardware circuit.

[0010] In yet another example, each of the cells is configured to perform one or more of a plurality of predetermined primitive operations in response to the one or more received instructions. The hardware circuit further includes a plurality of control cells. In transmitting the respective second instructions to the respective processing cells, the hardware circuit is configured to generate and transmit, by each control cell, a respective control signal to each processing cell based on a first operation specified by the first instruction. In yet another example, in generating and transmitting the respective control signal by each control cell, the hardware circuit is configured to generate a respective control signal for each processing cell to perform one of a respective arithmetic operation, a comparison operation, and a bypass operation based on at least one of a stage in which the processing cell resides or a data processing lane passing through the processing cell. In yet another example, the plurality of cells and the plurality of crossbars form a processing network of connected cells across the plurality of stages and the plurality of data processing lanes, the processing network of connected cells being configured to receive input data and generate respective output data in accordance with performing a first operation on the input data. In yet another example, a processing network of connected cells is configured to perform a combined vector sort and duplicate count operation, the combined operation including the processing network receiving an input vector of elements and the processing network generating as an output a sorted output vector and data specifying a count of duplicate elements in the input vector.

[0011] In yet another example, the input data includes sparse vector data, and after transmitting the respective second instruction and the respective third instruction, the hardware circuit is configured to perform one of a vector scan, a vector sum, a vector sort, or a vector duplicate count.

[0012] Yet another aspect of the disclosure provides a computer-implemented method, the method including a hardware circuit including a plurality of stages, each stage including a crossbar and two or more cells, and including a plurality of data processing lanes for streaming respective data from an upstream input to a downstream destination via the plurality of cells and the plurality of crossbars of the plurality of stages, receiving input data from an upstream input along the plurality of data processing lanes and a first instruction for performing a first operation. The method further includes, in response to receiving the first instruction, for each stage, the hardware circuit transmitting a respective second instruction to a respective processing cell of that stage, each cell being configured to perform a respective second operation in response to receiving an input from the respective data processing lane, and the method includes the hardware circuit transmitting a respective third instruction to a respective crossbar for that stage, the crossbar being configured to permute an output from each cell of that stage to a cell of a next stage along the plurality of data processing lanes. The method also includes the hardware circuit performing the first operation by processing the received input data along the plurality of data processing lanes and the plurality of cells configured to perform the respective second operations.

[0013] In one example, each cell is configured to receive a respective first input operand from a respective data processing lane passing through the cell and a respective second input operand from a respective crossbar of an upstream stage of the cell. In another example, a downstream destination of data for the multiple data processing lanes is a vector processing unit, the vector processing unit configured to perform a single instruction multiple data vector operation on output data of the hardware circuit.

[0014] In yet another example, each of the cells is configured to perform one or more of a plurality of predetermined primitive operations in response to the one or more received instructions. The hardware circuit further includes a plurality of control cells. Sending the respective second instructions to the respective processing cells includes generating and sending, by each control cell, a respective control signal to each processing cell based on the first operation specified by the first instruction. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of an example hardware circuit for implementing a cross-lane processing unit (XPU) in accordance with an aspect of the disclosure. [Diagram 2] 1 is a block diagram of an example tile for executing an XPU according to an aspect of the disclosure. [Diagram 3] FIG. 2 is a block diagram of an example XPU according to an aspect of the disclosure. [Figure 4] 1 is a block diagram of an example XPU processing cell and corresponding control cell according to an aspect of the disclosure. [Diagram 5] FIG. 1 is a block diagram of an example XPU configuration for vector scan operations performed by the XPU in accordance with an aspect of the disclosure. [Figure 6] 1 is a block diagram of an example XPU configuration for vector sort and vector duplicate count operations performed by an XPU in accordance with an aspect of the disclosure. [Figure 7] FIG. 13 is a block diagram of an example XPU configured to include an odd-even merge network and a value shuffle network in accordance with an aspect of the disclosure. [Figure 8A] 1 is a flow diagram of an example XPU configuration of an odd-even merge sort network for performing a combined merge sort and duplicate count in accordance with an aspect of the disclosure. [Figure 8B] FIG. 8 illustrates an example stage 1 of an odd-even merge network 800 of an XPU in accordance with an aspect of the disclosure. [Figure 8C]FIG. 1 illustrates an example stage 2 of an odd-even merge network of an XPU in accordance with an aspect of the disclosure. [Figure 8D] A diagram illustrating an example stage 3 of an odd-even merge network of an XPU in accordance with an aspect of the disclosure. [Figure 8E] A diagram illustrating an example stage 4 of an odd-even merge network of an XPU in accordance with an aspect of the disclosure. [Figure 8F] A diagram illustrating an example stage 5 of an odd-even merge network of an XPU in accordance with an aspect of the disclosure. [Figure 8G] FIG. 1 illustrates an example stage 6 of an odd-even merge network of an XPU in accordance with an embodiment of the disclosure. [Figure 8H] FIG. 1 illustrates an example value shuffle network for an odd-even merge network of an XPU in accordance with an aspect of the disclosure. [Figure 9] 1 is a flowchart of an example process for performing compositing operations in an XPU in accordance with an aspect of the disclosure. [Figure 10] 1 is a flowchart of an example process for sorting and generating duplicate counts for input vectors to an XPU in accordance with an aspect of the disclosure. [Figure 11] FIG. 1 is a block diagram of an exemplary environment for executing hardware circuitry according to an aspect of the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Detailed Description Summary Aspects of the disclosure are directed to a cross-lane processing unit (XPU) for performing data-dependent operations across multiple data processing lanes of a processor. Rather than executing physically fabricated operation-specific circuitry for each data-dependent operation, the XPU can be configured to perform different operations in response to input signals by configuring the individual operations to be performed by the processing cells and arranging the crossbars as a stacked network in the XPU. The XPU operates across values ​​of multiple SIMD data processing lanes. The XPU can be executed as part of a coprocessor configured for SIMD parallel processing. The coprocessor executing the XPU can be configured to perform the data-dependent operations.

[0017] Data-dependent operations, also referred to as "input-dependent operations," are operations in which the amount of computational effort to perform the operation is not known in advance and depends on the nature of the data. The computational effort can be measured, for example, in the number of operations or processing cycles required to perform the data-dependent operation. Exemplary data-dependent operations include operations for vector sorting, operations for identifying unique values ​​in an input vector, operations for counting duplicate values ​​in a vector, and operations for manipulating the shape or size of vectors of varying length. Data-dependent operations are irregular, at least due to differences in random memory access patterns for performing the same type of operation on different inputs. As a result, data-dependent operations are difficult to optimize for performance, as opposed to other types of operations in which the computational effort does not change based on the nature of the input data, such as its shape or degree or sparsity.

[0018] Data-dependent operations include operations performed on sparse data. The sparsity of a data structure is a measure of its ratio of non-empty elements to empty elements. Depending on the data structure, an empty element may be zero, a reserved word indicating the absence of a value for an element, or may have a value so small that it is considered to contribute insignificantly to the operations performed by the data structure as input. A data structure is sparse if it has more empty elements than non-empty elements. Some data structures can be more or less sparse than others.

[0019] Aspects of the disclosure provide an XPU for first processing sparse data before sending the data to downstream coprocessors in the processing pipeline, enabling a wider range of workloads for efficient computation than was previously possible without the XPU.

[0020] Because the XPU can handle a variety of data-dependent operations, processing pipelines and corresponding processors can be designed without the limitations of predefining input data for processing in existing SIMD architectures. Without the XPU, existing SIMD architectures cannot efficiently accelerate data-dependent operations, such as generating embeddings from a sparse set of features into a machine learning model.

[0021] Exemplary data-dependent operations include generating embeddings for input training examples. The embeddings can be vectors with higher dimensionality than the embeddings or some other data structure mapped from the input. The embedding generation can be performed as part of the workload processed according to the pipeline. As other examples, the XPUs may perform vector scatter or gather operations, segment sums, and / or partition sparse feature tensors. The XPUs described herein can be complementary processing units to other components or connected components of a processor, such as vector processing units built according to a SIMD parallel processing paradigm. One or more XPUs can be connected in each processor core of a larger processor, which itself may include other components to speed up the performance of a given workload, such as training a neural network.

[0022] Furthermore, the XPU is not limited to performing a given type of data-dependent operation, and thus the processor can be designed to include an XPU to complement other types of processing units for multiple different pipelines. Since the XPU can be configured per workload, the physical footprint of the XPU is reduced relative to other approaches where specialized circuits are physically fabricated on the processor as complementary units for computation of sparse data. The functionality of the XPU can also be extended through the use of instruction sets or extensions to the existing instruction set of the host processor, further improving the adaptability of different data-dependent operations as the pipeline data undergoes changes. Instructions can be provided as signals to the components of the XPU responsible for translating the instructions to configure the individual processing cells and crossbars of the XPU. The XPU can be configured using a program compiled by a corresponding compiler for the hardware circuitry that executes the XPU.

[0023] The XPU includes a network of individual processing cells, each cell processing data passing through one or more data processing lanes via crossbar connections between the processing cells. Each data processing lane may include one or more registers for temporarily storing data during processing. Each processing cell is configured to perform one or more primitive operations on a plurality of sets of operands. A first set of operands is provided as input from a data processing lane of a processor shared by the processing cells. A second set of operands is provided from a crossbar configured to coordinate data transmission across the plurality of data processing lanes of the XPU.

[0024] The XPU can be divided into multiple pipeline stages, each of which includes a crossbar, one or more processing cells, and a corresponding control cell for each processing cell. The number of stages can vary based on, for example, the compositing operations that the XPU is configured to perform for a current workload.

[0025] The XPU performs compositing operations by executing a plurality of primitive operations across the pipeline stages of a stacked network of processing elements and crossbars. Compositing operations are operations performed by the XPU on inputs to generate outputs. Primitive operations are operations that individual processing cells of the XPU are configured to perform that, when performed by the XPU, cause the XPU to perform a compositing operation. Performing a compositing operation may require performing other compositing operations. For example, to perform a vector sort, the XPU may perform a prefix sum, another operation comprised of a plurality of primitive operations. Exemplary primitive operations include comparison, arithmetic, or operations to bypass input data. The XPU performs compositing operations by configuring each of a plurality of individual processing cells and crossbars arranged according to one of a plurality of pipeline stages for the XPU.

[0026] The primitive operations performed in each stage of the XPU can be programmatically defined and may vary from workload to workload. The primitive operations that a processing cell is configured to perform are determined by one or more control signals or instructions received by a respective control cell for the processing cell. The exact primitive operations performed by a processing cell may depend, for example, on the compositing operation that the XPU is currently configured to perform. In other examples, processing cells in different lanes or different stages of an XPU may be configured to always perform one or more predefined primitive operations. After the XPU generates an output, the output may be sent along multiple data processing lanes to another processing unit or memory unit of the processor executing the XPU.

[0027] Aspects of the disclosure provide an XPU configured to receive an input vector and determine both unique values ​​in the vector and the number of duplicates for each value using the same stacked network configuration. The XPU can perform a vector sort, which is an in-place stable sort of the key-value tuples of the input vector, sorted by key. A vector duplicate count returns a running duplicate count of the values ​​of the key-value tuples of the input vector. The sorted list and duplicate counts can be used to remove non-unique values ​​from the input vector. The XPU is configured to perform both the vector sort and the duplicate count according to the same configuration of processing cells and crossbars as described herein. By using the same configuration, the XPU can perform both compositing operations more efficiently, since at least the XPU does not have to be reconfigured between performing a vector sort to identify unique values ​​and generating a vector duplicate count for a given input vector. Other compositing operations that the XPU is configured to perform include scan, segmented scan, parallel-prefix sum, vector partition, vector histogram, vector compact, vector replace, vector reduce, vector shift-insert, vector gather, vector scatter, etc.

[0028] Aspects of the disclosure may provide the following technical advantages: Hardware circuitry executing the XPU may provide more flexible and programmable hardware for embedded class workloads and other data-dependent operations that are not efficiently parallelizable. The XPU provides an acceleration path for different classes of data-dependent operations for different workloads, without requiring the XPU to be fixed to efficiently execute only certain operations. By providing programmable units as described herein, the execution hardware circuitry can reliably adapt to the demands of different workloads, complementing parallelizable data-independent SIMD operations that may otherwise be inefficient or ineffective for workloads requiring data-dependent operations.

[0029] Hardware circuits such as application specific integrated circuits can be designed with different amounts of XPUs to further coordinate and distribute the workload at scale. The XPUs described herein also enable efficient execution of multiple operations using the same configuration, further reducing processing and configuration time. For example, an XPU can be configured to perform both vector sorting and vector duplicate counting, instead of separate configurations of the XPU and / or separate instances of specialized circuitry to speed up these operations.

[0030] Exemplary System 1 is a block diagram of a hardware circuit 101 executing cross-lane processing units (XPUs) 101A-101F according to an embodiment of the disclosure. The hardware circuit 101 may include a coprocessor 103, a coprocessor 104, a high-bandwidth memory 107, and an on-chip interconnect 108. The coprocessor 103 may include one or more tiles 102A-102F, each executing a respective XPU 101A-101F. The coprocessor 103 may include a tile sequencer 106 configured to coordinate input and output data across the tiles 102A-102F.

[0031] Coprocessor 103 is configured to execute data-dependent operations using XPUs 102A-102F. As described in more detail herein with reference to Figures 2-4, XPUs 102A-102F are connected to a series of data processing lanes, each lane streaming data through XPUs 102A-102F. The stream data can be retrieved from on-chip memory 105, which can be any of a variety of different memory devices including main memory, cache, or persistent storage, e.g., solid-state or hard disk storage. The stream data can also be retrieved from coprocessor 104, high-bandwidth memory 107 serving one or both of coprocessors 103 and 104, and / or another data source connected to hardware circuit 101 via on-chip interconnect 108.

[0032] Coprocessor 104 can be configured for acceleration of certain operations, such as matrix-matrix multiplication, matrix-vector multiplication, etc. In some examples, coprocessor 104 is on a different device than the rest of hardware circuitry 101 and communicates data to the hardware circuitry via on-chip interconnect 108. On-chip interconnect 108 can be a data bus or any type of interconnect according to any of a variety of communication standards, e.g., PCIe.

[0033] Exemplary inputs to the hardware circuit 101 are tensor-represented input data and / or model parameters of a machine learning model executed using the hardware circuit 101. Tensors are data structures that generalize various other common data structure types of different dimensions. Tensors can contain zero or more elements, which can be one or more different data types, such as integers, floating-point values, Boolean values, etc. Within each data type, the data type can be parameterized according to a given level of precision, e.g., 8-bit, 16-bit, or 32-bit integer or floating-point values. The dimension of a tensor is called its "rank." A rank-zero tensor is a single element and is also called a scalar. A rank-one tensor is also called a vector. A rank-two tensor is also called a matrix. Vectors and matrices can also be referred to as having different ranks. For example, a rank-two vector is equivalent to a matrix. A non-zero rank tensor can be described as a collection of tensors one rank lower. For example, a rank-1 vector is a collection of scalar values, and a rank-2 matrix is ​​a collection of rank-1 vectors.

[0034] The hardware circuit 101 may at least partially execute a processing pipeline for training a neural network. A neural network is a machine learning model that includes one or more layers of nonlinear operations to predict an output for a received input. In addition to an input layer and an output layer, some neural networks include one or more hidden layers. The output of each hidden layer can be an input to another hidden layer or an output layer of the neural network. Each layer of the neural network can generate a respective output from a received input according to values ​​for one or more model parameters for that layer. The model parameters can be weights or biases that are determined via a training algorithm to cause the neural network to generate an accurate output. A feature is a measurable property of an input example received by a machine learning model. A feature value is a value for each feature and can be an integer value.

[0035] The pipeline may include generating an embedding for the input training examples. The embedding may represent features of the input training examples using less data, allowing for more efficient processing. The feature tensors for different input training examples have different degrees of sparsity, which affects the amount of computational work required to generate the corresponding embeddings. The hardware circuitry 101 may be configured to receive a tensor of feature values ​​representing the training input examples and generate an embedding as a tensor having a lower rank than the feature tensor.

[0036] To generate the embeddings, coprocessor 103 is configured to perform various data-dependent operations for efficient sparse data computation in XPUs 102A-102F. These operations include sorting or summing sparse vectors, operations for summarizing the contents of input vectors, and operations for converting sparse matrices from one sparse matrix storage format to another.

[0037] Instead of physical predefined circuitry to speed up performance of data-dependent operations, XPUs 102A-102F can be configured, e.g., programmed, to perform a variety of different data-dependent operations. Coprocessor 103 allows generalized support for processing sparse data while still allowing complementary coprocessor 104 to perform highly efficient and parallelizable operations.

[0038] The hardware circuitry 101 can be any of a variety of different types of processing units, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC) such as a tensor processing unit (TPU). The hardware circuitry 101 can be executed on a computing device, which itself may be part of a system of one or more devices, as described herein with reference to FIG.

[0039] 2 is a block diagram of an exemplary tile 102 executing an XPU 101. The XPU 101 is coupled to a cross lane controller 210. The cross lane controller 210 provides separate threads of control to allow cross lane instructions on the XPU 101. As described herein, the XPU can receive a first instruction, for example, via one or more control signals, which can be translated into one or more second and third instructions and provided to the processing cells and crossbars of the XPU 101, respectively, to perform the compositing operation specified by the first instruction. The instructions to the XPU 101 can be carried on control signals, which the processing cells and crossbars of the XPU 101 are configured to interpret to perform a corresponding primitive operation. An exemplary instruction can be an opcode of an instruction set architecture (ISA).

[0040] The tiles 102 may receive data from the on-chip interconnect 108 and the on-chip memory 105, as described with reference to FIG. 1. The XPUs may also receive instructions via an instruction interface 224, e.g., from the tile sequencer 106, the scalar core 212 or the scalar core 220. The scatter / gather controller 222 of the tile 102 may receive the incoming data and control which data is passed through the memory scheduler 214 in the memory 206. The memory scheduler 214 coordinates how the data is evaluated and retrieved from the memory 206. The memory 206 is private to the tile 102 and is not accessible by other components connected to the tile 102, such as other tiles. The arbiter 204 is configured to manage which of the VPUs 202A-202H access the memory 206, e.g., in a clock-cycle-to-clock-cycle manner. The tiles 102 may maintain a task queue 208 of tasks to be executed by the tile 102. The tiles 102 may also maintain registers of tile synchronization flags 218 and / or memory flags 216 to synchronize the tiles 102 with other tiles in the hardware circuit 101 and memory 206, respectively.

[0041] Vector processing units (VPUs) 202A-202H are connected to XPU 101 via data processing lanes, indicated by solid lines between XPU 101 and VPUs 202A-202H. Dashed lines between XPU 101 and VPUs 202A-202H represent control signals that may be received by control cells in XPU 101 to configure XPU 101 to perform compositing operations corresponding to the received control signals. The vector processing units are configured for efficient operations on input vectors. The length of a vector processed by a tile 102 at one time may depend on the number or width of VPUs executed by the tile. For example, eight VPUs 202A-202H are eight wide. The VPUs 202A-202H may process data along the same data processing lane. The VPUs 202A-202H may be configured to perform scalar operations on elements of received vectors from memory 206. VPUs 202A-202H can receive data from XPU 101, and XPU 101 can process the data across data processing lanes, rather than simply along the lanes as performed by each VPU 202A-202H, as described herein.

[0042] FIG. 3 is a block diagram of an exemplary XPU 300. XPU 300 includes processing cells 301-309, crossbars 203-310, and control cell 350. Control cell 350 is represented by a shaded block in the block diagram of FIG. 3. Data flows from bottom to top along data processing lanes 300A-300H, beginning at stage 1 and ending at stage 6. Stage 1 includes processing cell 301 and crossbar 302. Stage 2 includes processing cell 303 and crossbar 304. Stage 3 includes processing cell 305 and crossbar 306. Stage 4 includes processing cell 307 and crossbar 308. Stage 5 includes processing cell 309 and crossbar 310. Stage 6 includes processing cell 311 and crossbar 312. In different examples, the XPU can include more or fewer stages. The XPU can also include a crossbar 399.

[0043] For purposes of explanation, earlier stages are considered "upstream" with respect to later stages, and later stages are considered "downstream" with respect to earlier stages. For example, stage 1 is upstream of stage 5, stage 4 is downstream of stage 3, etc.

[0044] The crossbar in each stage of the XPU can be any type of circuitry configured to permute different input values ​​from each lane to different other processing lanes according to the current configuration of the crossbar. The crossbar can receive one or more control signals from the control cells for each processing cell in the same stage as the crossbar. The crossbar is configured to permute input values ​​from each processing cell in the same stage according to a fixed pattern. The pattern depends on the compositing operation the XPU is currently configured to perform and does not necessarily cause the crossbar to permute all processing cell outputs. In other words, some processing cell outputs may bypass the crossbar and proceed to the next stage along the same processing lane.

[0045] To configure the processing cells, each processing cell of XPU 300 has a respective control cell 350 configured to receive one or more control signals along the respective processing lane in which the processing cell resides. As described in more detail with reference to FIG. 4, the processing cells are configured with circuitry for performing a variety of different primitive operations and for performing these operations in accordance with received control signals or instructions. The control cells receive instructions along the data processing lanes, for example, as one or more signals that can be interpreted by the control cell to determine which primitive operation its corresponding processing cell will perform. The control cells can forward the control signals to the processing cells, or process the received instructions or signals and forward generated control signals that the processing cells are configured to receive to enable or disable execution of the specified primitive operations.

[0046] A processing cell may also be configured to bypass input data received from the respective processing lane for the processing cell. When bypassed, the received input is sent without modification from the processing cell to the crossbar in the same stage as the processing cell. Inputs received by a bypassing processing cell from the crossbar of a previous stage may be tied to zero or ignored. The actual behavior of a bypassing processing cell may depend on the pipeline stage in which the cell resides and / or the processing lane in which the processing cell resides. Figure 4 illustrates an exemplary processing cell configured to perform a comparison operation, an arithmetic operation, and / or a bypass primitive operation.

[0047] XPU 300 may be configured to receive instructions defined as part of an instruction set architecture or an extension to an instruction set architecture that the processor executing XPU 300 is configured to apply and execute. The instructions may specify different compositing operations and / or primitive operations that the XPU and the individual processing cells are configured to execute as corresponding operations. Control cell 350 is configured to receive data representing instructions defined as part of an instruction set or an extension and / or to convert the instructions into control signals to configure the corresponding processing cells. For example, control cell 350 may receive signals as opcodes, codewords for operations that the XPU is configured to execute of an instruction set corresponding to the processor or hardware circuit executing XPU 300. When XPU 300 receives an instruction to perform a compositing operation, such as vector sorting or vector duplicate counting, XPU 300 may configure each processing cell to execute a predetermined respective primitive operation that causes the XPU to execute the instructed compositing operation.

[0048] The operations performed by the XPU may be synchronized by clock cycles. For example, the operations performed by the processing cells in each stage may be performed in one or more cycles. For example, the operations in each stage may be performed in a single cycle. Different compositing operations performed by the XPU may take different amounts of clock cycles to execute. For example, vector sorting may be performed by the XPU in six cycles, vector prefix sum in four cycles, and vector compact in two cycles.

[0049] As described in more detail with respect to Figure 4, the processing cells may be configured to perform arithmetic operations, such as addition, between operands of different types, including floating-point values ​​and signed or unsigned integers. Arithmetic operations, such as addition, may form part of the compositing operations performed by the XPU for scanning operations, as described herein with reference to Figure 5.

[0050] Exemplary instructions include instructions for resetting the XPU and retrieving information about clock synchronization and compositing and primitive operations performed by the XPU. Other instructions include instructions for retrieving one or more of operands, mask values, and / or segment markers from each processing lane. Instructions may include instructions for accessing data structures stored by the XPU along with control information specifying each of the various different compositing operations supported by the XPU. In yet another example, instructions may include instructions for causing the XPU to push data into various registers, latches, or flip-flops and determining whether the contents of the foregoing are valid. The pushed data may include, for example, values ​​to be processed as part of performing a compositing operation, and / or mask values.

[0051] A configured XPU 300 is said to execute a processing network to perform a particular compositing operation. For example, XPU 300 includes 48 XPU cells that can be configured as follows: 18 cells can be configured for arithmetic operations, 38 cells are configured to compare input values ​​(one cell may be configured for both arithmetic and comparison operations), and 10 cells are configured to bypass inputs. In response to a new instruction, XPU 300 can reconfigure itself with a new processing network to perform a different compositing operation.

[0052] The XPU can be configured to operate in a variety of different operational modes, which may be specified as different instructions in the instruction set or extensions. The different operational modes may include different compositing operations to sort, count duplicates, identify unique values, scan, partition data, and / or identify unique values ​​in data input to the XPU. Additionally, the instructions may include operands that specify the type of comparison or arithmetic operation, for example, to perform an unsigned integer compare for sorting or scanning, or a floating point add. Other operands to an instruction to perform a compositing operation include specifying from which processing lane the output of the compositing operation emerges from the XPU 300. Other operands received may include segment markers to perform a compositing operation on a segment of input data, for example, to sort each of multiple segments of data received by the XPU 300 across data processing lanes.

[0053] When performing vector sorting and / or vector duplicate counting, the XPU 300 is configured to include an odd-even merge network and a value shuffle network. Exemplary network configurations are shown and described with reference to Figures 4-8G. The network configuration includes one or more stages of XPUs, with each cell and crossbar of each stage configured to perform one or more primitive operations. After sorting the vectors, the XPU 300 can identify and remove non-unique values ​​in the input vectors, leaving only unique values, e.g., values ​​that occur only once in the processed input vector.

[0054] XPU 300 may include register files 360A and 360B. Register files 360A and 360B are coupled to data processing lanes 300A-H between different stages and may be used to store and retrieve data. For example, some data may be stored in register file 360B after processing cell 307 in stage 4, while data output by XPU 300 is stored in register file 360A.

[0055] 4 is a block diagram of an exemplary XPU processing cell 400 and a corresponding control cell 499. The processing cell 400 can be configured to perform a number of primitive operations as described herein based on control signals. The control signals can represent opcodes that form part of an instruction set for programming the XPU.

[0056] XPU processing cell 400 is one example of a possible circuit implementation of an XPU processing cell. In different examples, XPU processing cell 400 may include more or fewer components for implementing circuitry configured to perform different primitive operations. In some examples, XPU processing cell 400 may include circuitry configured for only a subset of the primitive operations. In other examples, XPU processing cell may include circuitry configured to perform all of the primitive operations in sequential or parallel fashion.

[0057] Each processing cell can also receive an input tuple of values. The passed values ​​can be individual values, such as integer or floating point values, or groups of values, such as tuples or lists of integer or floating point values.

[0058] The source inputs include x_source_1 402 and y_source_1 402, collectively the "source inputs." The source inputs may include elements of a vector to be processed by the XPU as described herein. The source inputs may pass through one or more of a 32-bit floating point adder (FP32 ADD) 403, a 32-bit signed integer adder (S32 ADD) 404, a 32-bit floating point comparator (FP32 405), and a 32-bit signed integer comparator (U32 CMP) 406. The adders 403, 404 may calculate a sum of the source inputs and send the output sum to a multiplexer 407. The comparators 405, 406 compare the input values ​​and send the max / min value as an output to a comparison selector 492.

[0059] A comparison selector 492 may receive the outputs of comparators 405, 406, and mask values ​​x_mask 444 and y_mask 446. The mask values ​​may be used as part of certain compositing operations described herein to mask some or all of the inputs received by the processing cell 400.

[0060] The comparison selector 492 selects an input and sends the input to the multiplexer controller 440 as selection_output 472. The comparison selector 492 may also receive an output from a comparator (CMP 426). The comparator 426 may receive the lane identifiers x_lane_id 422 and y_lane_id 424 and compare the lane identifiers to determine the greater of the two. The comparator 426 may also send its output to the multiplexers 472, 476, 478 and / or 480. The output from the comparator 426 may be a Boolean value (referred to as x_lane_id_greater_y_lane_id 490) indicating whether the lane identifier for the x lane is greater than the lane identifier for the y lane. The selection_output 472 may be determined based on the configuration of the comparison selector 492, as described herein with reference to FIG. 8A.

[0061] Multiplexer 407 may receive the sum of adders 403, 404 and / or the source inputs 401, 402 directly (e.g., if processing cell 400 is configured to perform a bypass). The output of multiplexer 407 may be stored in register 408 and sent as an output to the processing cell as y_source_1_out 484. Multiplexer 407 may also receive the output lane identifier into which the "x" data is sourced as out_lane_id_pre 488.

[0062] Multiplexers 472 and 476 may receive values ​​representing the duplicate counts as x_dup_count 414 and y_dup_count 416. As described herein with reference to FIG. 8A, processing cell 400 may be configured to calculate the duplicate counts of values ​​of an input vector, for example, while sorting the input vector. Increment values ​​x_increment 418 and y_increment 420 may also be received by multiplexers 478 and 480, and are values ​​used as part of performing duplicate counting with an odd-even merge network according to aspects of the disclosure and as described herein with reference to FIG.

[0063] The values ​​430, 432, 434, and 436 are sent to a lane count calculator 428. The lane count calculator 428 can use a value representing the larger lane count of the input lanes (larger_lane_count 430), a value for the smaller lane count of the input lanes (smaller_lane_count 432), the larger of the increment values ​​418, 420 (larger_lane_incr 434), and the smaller of the increment values ​​418 and 420 (smaller_lane_incr 436) to calculate a count of overlapping values ​​across the input data processing lanes x and y. An exemplary implementation of the lane count calculator 428 is described herein with reference to FIG. 8A and the "MAX" cell.

[0064] The XPU processing cell 400 may generate multiple outputs that are sent to the crossbar of the same stage as the processing cell 400. A register 482 may hold an output value (y_source_2_out 486), e.g., a bypassed input value from processing lane y. The mask values ​​444, 446 may be passed through a logical OR circuit 442, the output of which may be stored in a register 487. The processing cell may output the mask value for the processing cell as y_mask_out 482.

[0065] The multiplexer 448 may receive a default value of 0 and the smaller_lane_count 432 from the lane count calculator 438, and may output an increment value (y_incr_out 458). The output to the multiplexer 448 may be stored in a register 454. The multiplexer 452 may receive a default value of 0 and the duplicate counts 414, 416, and may output a count value for the processing cell 400 (y_count_out 460). The output to the multiplexer 452 may be stored in a register 454. The multiplexer 456 may receive the lane identifiers 422 and 424, and may output a lane identifier (y_lane_id_out 462). The output to the multiplexer 456 may be stored in a register 458.

[0066] 4 indicate control signals passed through processing cell 400 and control cell 499. Control cell 499 may receive instructions, for example as opcode 428. Control cell 499 sends instructions, for example as control signals, to multiplexer controller 440 and lane count calculator 438. Multiplexer controller 440 is configured to send signals to multiplexers 407, 448, 452, and 456 based on the received instructions. The output of lane count calculator 438 is also based on the received signals from control cell 499. The instructions sent by control cell 499 to XPU cell 400 may depend on the stage and lane of XPU cell 400.

[0067] Each of multiplexers 407, 448, 452 and 456 may be configured according to instructions received, for example as control signals, from multiplexer controller 440. In response to the received signal, multiplexers 407, 448, 452 and 456 output a value different from the received input.

[0068] In addition to sending instructions to the processing cell 400, the control cell 499 may also send the opcode 428 downstream (as opcode_out 468) and send instructions (as xbar_ctrl_out 470) to downstream crossbars in the same stage as the control cell 499. The instructions sent to the crossbar may cause the crossbar to permute received inputs according to a fixed pattern. Each control cell of the XPU may receive a different opcode and configure a corresponding processing cell to perform one or more primitive operations as described herein. The configured processing cells and crossbars form a processing network for performing compositing operations, examples of which are provided herein with reference to Figures 5 and 6.

[0069] 4, in some examples, processing cell 400 may use any of a variety of circuits for temporarily storing data, such as from a multiplexer or arithmetic or comparison circuitry. A register may include one or more flip-flops or latches.

[0070] Exemplary Methods Vector Scan and Vector Segmented Scan 5 is a block diagram of an example XPU configuration 500 for a vector scan operation performed by an XPU, according to an aspect of the disclosure. A vector scan is a rolling sum of elements in an input vector. An XPU when configured using configuration 500 can output a vector in which each element is a rolling sum of elements in the input vector up to the exponent of the element. For example, the first element of the output vector is equal to the first element of the input vector. The second element of the output vector is equal to the sum of the first element of the input vector and the second element of the input vector. The third element of the output vector is equal to the sum of the first, second and third elements of the input vector, etc.

[0071] In addition to vector scan, the configuration 500 can also be used for vector segmentation scan. The configuration 500 receives a set of mask bits to determine a segment of a vector. The XPU cells are configured to receive the mask bits and to bypass or transform data to downstream XPU cells depending on whether the XPU cells receive an input element in the same segment as the value of the respective XPU cell receiving the mask bit.

[0072] Configuration 500 shows a crossbar that substitutes output tuples, as shown in FIG. 4, e.g., y_source_1_out, y_mask_out, y_incr_out, y_count_out, and y_lane_id_out, from different lanes of a previous stage, with input tuples, e.g., x_source_1, x_mask, x_incr, x_count, and x_lane_id, in the current lane. In FIG. 5, the dotted lines represent data streamed through processing lanes 500A-500H. The solid lines represent how data is replaced between stages in configuration 500. Configuration 500 is a four-cycle configuration. When implemented on an XPU with five or more crossbars, the crossbars in stages not used as part of configuration 500 can be configured to bypass data through their respective lanes.

[0073] Beginning at stage 505, the input element in lane 500A is bypassed and each element in lanes 500A-500G is sent to a respective cell 501B-501H. Cell 501A bypasses the input element from lane 500A to the next stage. Cells 501B-501H sum the input element from the shifted lane with the input element of the lane in which each respective cell resides. The output of each cell 501A-501H is sent downstream.

[0074] In stage 510, the input elements of lanes 500A-500E are transposed from crossbar 505A to their respective lanes two lanes away from the input element's lane. Cells 502A and 502B bypass the received inputs of the next stage downstream. Cells 502C-502G sum the input elements of their respective lanes with their respective transposed input elements. The output of each of cells 502A-502H is sent downstream.

[0075] In stage 515, cells 503A-503C bypass their inputs to register 523 in each cell's respective lane. Cells 503D-503H receive shifted inputs from crossbar 510A and from three lanes away for each lane. Cells 503D-503H generate the sum of their respective lane's input elements and the shifted inputs they received. The output of each of cells 503A-503H is sent downstream into register 523 and then sent as part of the XPU's output vector.

[0076] In stage 520, the output data from stage 515 may be temporarily stored in a register file 523 before being sent as a downstream output of the XPU.

[0077] Vector sort and vector duplicate count 6 is a block diagram of an example XPU configuration 600 for vector sort and vector duplicate count operations performed by an XPU in accordance with an embodiment of the disclosure. Configuration 600 has six stages 605-630.

[0078] To perform a vector sort, the XPU can receive tuples of keys and values. An example input is: key={5, 5, 10, 1, 512, 8, 1023, 650} Value = {a, b, c, d, e, f, g, h} It can be.

[0079] An example of the output from the XPU to perform a vector sort on the example input above is: Key={1, 5, 5, 8, 10, 512, 650, 1023} Value = {d, a, b, f, c, e, h, g} In the event of a join, the key sourced from the higher numbered VPU lane is selected as the larger of the combined values. For example, in the event of a join between lanes 0 and 1, the key from lane 1 is selected as coming from the higher numbered VPU lane.

[0080] To perform vector duplicate counting, an XPU can receive a vector of values ​​and produce as output a vector where each element corresponds to the number of times a unique value appears in the input vector up to that element.

[0081] An example input is: Value = {10, 5, 10, 5, 10, 5, 5, 10} It can be.

[0082] Example output is: Count = {1, 1, 2, 2, 3, 3, 4, 4} It can be.

[0083] As described in more detail with reference to FIG. 7 and 8A-8H, in each stage, the processing cells are configured as MIN or MAX cells and form part of an odd-even merge sort network and a value shuffle network. The configuration 600 is a six-stage configuration. If the XPU executing the configuration has more than six crossbars, the remaining crossbars can be configured to bypass input elements. According to disclosed aspects, the same XPU configuration 600 allows for sorting vector elements while generating a count of duplicate elements for each vector. The streaming of data across the cells and crossbars of an XPU configured for vector sorting and duplicate counting is described with reference to FIG. 7 and 8A-8H.

[0084] The resulting output vector can be further processed to remove non-unique instances of a value. Given the duplicate count and the sorted list, the XPU can iterate through the list and remove non-unique values, producing an example output such as: Eigenvalues ​​= {1,2,3,4} Because the XPU configuration 600 enables both sorting and duplicate counting, the XPU can further generate a "uniqued" output vector that contains only a single instance of each value present in the input vector.

[0085] 7 is a block diagram of an exemplary XPU 700 configured to include an odd-even merge network 705 and a value shuffle network 710. Key and value tuples pass through VPU lanes 700A-N and enter odd-even merge network 705, where the keys are sorted. XPU 700 includes eight data processing lanes, although in other examples XPU 700 may have fewer or more data processing lanes, e.g., a number of data processing lanes equal to a power of two. In these examples, odd-even merge network 705 and value shuffle network 710 are correspondingly expanded or condensed without loss of generality.

[0086] Values ​​do not travel with the keys in sorted network 705, as indicated by the dotted lines through sorted network 705, but instead pass directly to value shuffle network 710 to be rearranged with their corresponding keys. Although the elements of the input vector are described as key-value tuples, it will be understood that any type of data that can be compared, such as integer values, floating point values, etc., can be received as input to the XPU.

[0087] In some examples, an odd-even merge sort of batchers can be performed in an odd-even merge network 705. As described herein with reference to FIG. 8A, the odd-even merge network can include multiple stages for processing input key-value tuples and sorting the tuples by key.

[0088] After the tuples are sorted by key, the value shuffle network 710 shuffles the values ​​to the lanes corresponding to their associated keys. The value shuffle network 710 can be an 8×8 32-bit crossbar. The inputs to the crossbar are 32-bit values ​​and lane identifiers. The value shuffle network 710 can support multiple modes depending on the lane identifier. For example, the value shuffle network 710 is configured to route the 32-bit value from the corresponding source lane when a source lane identifier is provided. In another mode, the destination lane identifier is used to route the 32-bit value to the appropriate output lane. For vector sorting, the value shuffle network 710 can be set to a source lane identifier mode and the value shuffle network 710 is set to a destination lane identifier mode while in vector duplicate counting, as described herein. For vector duplicate counting, the value shuffle network 710 shuffles the count values ​​with their corresponding values. The operation mode can be specified as one or more signal carrying instructions provided by the control cell of the XPU.

[0089] FIG. 8A is a flow diagram of an example XPU configuration 800 of an odd-even merge sort network 705 for performing a combined merge sort and duplicate count. The example XPU has eight processing lanes, labeled 0 through 7. The odd-even merge sort network 705 has six stages. The six stages are stage 1 801A (1×1 odd-even merge), stage 2 801B (2×2 merge), stage 3 801C (2×2 odd-even merge), stage 4 801D (merge phase 1), stage 5 801E (merge phase 2), and stage 6 (odd-even merge) (respectively, “stage 1”, “stage 2”, “stage 3”, “stage 4”, “stage 5”, and “stage 6”).

[0090] At each stage, the processing cells of the XPU are configured to perform a primitive operation to determine the minimum of two input operands and return the smaller operand with the lane identifier of the data processing lane in which the smaller operand was received. At each arrow shown in Figures 8A-8H, the XPU includes a processing cell configured to compare the input operands. A processing cell configured to determine the minimum of the two input operands is called a MIN cell, while a processing cell configured to determine the maximum of the two input operands is called a MAX cell.

[0091] An example pseudocode implementation of a MIN cell is shown in Table 1. For purposes of this and other examples, the pseudocode may refer to the variables and input / output values ​​shown and described with reference to FIG.

[0092] [Table 1]

[0093] As shown in Table 1, x and y are values ​​passed through the corresponding lanes with lane identifiers x_lane_id and y_lane_id, respectively. For example, in stage 1, lane 0 can pass value x and lane 1 can pass value y. In line 1 of Table 1, the variable x_less_than_y_ordered is a Boolean value. The variable is true under two conditions. First, the variable is true when the values ​​x and y are equal to each other and the lane identifier for the lane with the x value is not greater than the lane identifier for the lane with the y value. This condition is set to break the tie when x is equal to y by setting the lane with the lower lane identifier as the minimum of the values ​​being compared. Second, the variable is true if the value x is less than the value y. If the variable x_less_than_y_ordered is true, then in lines 3-5, the MIN cell outputs x as the minimum of x,y and also outputs the x_lane_id lane identifier. Otherwise, the MIN cell outputs y and y_lane_id, as shown in lines 7-9.

[0094] A processing cell may also be configured to perform a primitive operation to determine the maximum of two input operands, called a MAX cell. In both primitive operations, one operand comes from a data processing lane for the respective processing cell, while a second operand is provided across the crossbar from another data processing lane.

[0095] An example pseudocode implementation of a MAX cell is shown in Table 2:

[0096] [Table 2]

[0097] In line 1 of Table 2, the variable x_greater_than_y_ordered is a Boolean value that is true when the value x is equal to the value y and the lane identifier for the lane corresponding to the value x is greater than the lane identifier for the lane corresponding to the value y, or when the value x is greater than the value y. The MAX cell outputs x and the lane identifier x_lane_id when x_greater_than_y_ordered is true, and outputs y and the lane identifier y_lane_id otherwise. The first condition is a tiebreaker when the value x is equal to the value y.

[0098] To configure the odd-even merge sort network to also obtain duplicate counts for the input elements, the MAX cells in the various stages are configured as described herein. The modifications to the MAX cells to perform duplicate counts do not substantially add timing and logic complexity, and in some examples can be enabled or disabled based on the current workload being processed by the XPU. By combining the vector sort and duplicate count operations as part of the same configuration of the XPU, the XPU can more efficiently obtain values ​​for both operations without separately configured hardware and / or without reconfiguring the XPU between operations. The MAX cells perform different operations depending on whether the MAX cells are part of an odd-even merge stage, where odd and even elements are compared and sorted, or part of an odd-odd / even-even merge, where odd elements are compared separately from even elements. In the following description, the term "value" is used generally to refer to the elements being sorted. As shown and described with reference to Figures 8B-8H, the values ​​being sorted can be keys of tuples of key-value pairs.

[0099] In the 1×1 odd-even merge stage 1, adjacent elements are compared to form four sorted lists of length 2, indicated by the four dashed boxes in stage 1. In the 2×2 odd-odd / even-even merge stage 2, the odd elements of adjacent sorted lists are compared. The even elements of adjacent sorted lists, indicated by the two dashed boxes in stage 2, are also compared. Stage 3 is a 2×2 odd-even merge stage where each i-th element is merged with the i+1-th element. Stages 4 and 5 are phases 1 and 2 of the 4×4 odd-odd / even-even merge. In stage 4, the first set of odd elements of adjacent sorted lists are compared. The first set of even elements are also compared. In stage 5, the second set of odd elements of adjacent sorted lists are compared. The second set of even elements are also compared. Stage 6 is a 4×4 odd-even merge phase where the i-th element is sorted with the i+1-th element. The output of stage 6 is a single sorted list of length 8. After sorting, the tuples sorted by key are passed through a value shuffle network to shuffle the counts computed by the XPUs to their corresponding keys. Each stage is described in more detail with reference to Figures 8B-8H.

[0100] In stage 1, the XPU compares the values ​​in lanes 0, 1, 2, 3, 4, 5, 6, and 7. The dashed blocks in stage 1 of FIG. 8A represent the four sorted lists after the XPU sorts the values ​​of pairs of adjacent lanes. If the value in the lower numbered lane is greater than the value in the higher numbered lane, the XPU swaps the values ​​between the lanes. For output lane l, the count C l is the sum of values ​​passing through the set of lanes that have the same value as the value in lane l. l contains lanes whose identifiers are less than or equal to the current lane. For example, the set S for lane 5 l may include lanes 1 to 4, but may not include lane 6.

[0101] To calculate the duplicate count, the MAX cell in stage 1 can be configured to set the count of the upper lane to one more than the count of the lower lane. The upper and lower lanes can be determined based on the VPU lane that sources the value whose count is to be incremented. An example pseudocode implementation for performing the sort and duplicate count in the odd-even merge stage is shown in Table 3 below:

[0102] [Table 3]

[0103] In Table 3, the MAX cell determines the Boolean value of the variable x_greater_than_y_ordered according to line 1. According to lines 3-9, the MAX cell first determines whether the x_lane_id identifier is greater than the y_lane_id for the lanes streaming the values ​​x and y, respectively. If the variable x_lane_id_greater_than_y_lane_id is true, the MAX cell sets the count_large variable to the count variable x_count for the lanes streaming the value x, and sets the count_small variable to the count variable y_count for the lanes streaming the value y. Otherwise, the variable count_large is set to y_count and the variable count_small is set to x_count, as shown in lines 7-9.

[0104] According to line 11, the MAX cell increments the value for the variable count_large. According to lines 13-21, based on the value for the variable x_greater_than_y_ordered, whenever the input values ​​x and y are equal, the incremented count_large variable is output (lines 16, 21), or else the respective count of the greater of the two values ​​is output. For example, if the predicate x_equal_y is not true in line 16 and x_greater_than_y_ordered is true, then out_count=x_count in line 16 is output. Similarly, if the predicate x_equal_y is not true in line 16 and x_greater_than_y_ordered is not true, then out_count=y_count in line 16 is output.

[0105] Stage 2 is a 2x2 odd-odd / even-even merge stage. Odd elements of adjacent sorted lists from the four sorted lists from stage 1 are compared and, if necessary, swapped. As shown in Figure 8A, the value in lane 0 is compared to the value in lane 2, the value in lane 1 is compared to the value in lane 3, the value in lane 4 is compared to lane 6, and the value in lane 5 is compared to lane 7.

[0106] In the odd-odd / even-even stage, count C for the even lanes (e.g., lane 0, lane 2, lane 4, etc.) l is a set of lanes l The set of lanes is the sum of the values ​​in lane l and v f , which is less than or equal to the current lane in even lanes.

[0107] To calculate the duplicate count for lane l in a 2x2 odd-odd / even-even merge stage, the MAX cell adds the counts of equal values ​​in the lanes being compared and assigns the sum to the upper lane. The lower and upper lanes are determined based on the lane that sourced the value being counted.

[0108] Stage 3 is a 2x2 odd-even merge stage. The XPU in this stage merges the two lists from the previous stage and compares the values ​​in lanes 1 and 2 with the values ​​in lanes 5 and 6. An example pseudocode implementation for performing sorting and duplicate counting in a 2x2 odd-even merge stage is shown below in Table 4.

[0109] [Table 4]

[0110] In Table 4, the variable count_large is the sum of the counts from lanes x and y being compared. Whenever the input values ​​x and y are equal (lines 16, 21), the MAX cell outputs the count_large variable, otherwise it outputs the respective count of the larger of the two values. For example, if the predicate x_equal_y is not true in line 16 and x_greater_than_y_ordered is true, then out_count=x_count in line 16 is output. Similarly, if the predicate x_equal_y is not true in line 16 and x_greater_than_y_ordered is not true, then out_count=y_count is output.

[0111] Stage 4 is the first phase for a 4x4 odd-even merge stage. The set of output lanes can be described as including a set of even lanes and a set of odd lanes. The first phase is similar to merging the input sorted lists as described herein for a 2x2 odd-odd / even-even merge sort.

[0112] Stage 5 is the second phase for a 4x4 odd-even merge. For example, as shown in FIG. 8A, the even lanes to be sorted and merged are lanes {2,4} and the odd lanes to be sorted and merged are {3,5}. The MAX cell is configured to manage and handle multiple duplicate counts that may otherwise be inadvertently added to the output lanes during a 4x4 odd-even merge. The duplicate count in lane 2 can be C6 from lane 6 or C2 from lane 2 (because lanes 2 and 6 are compared during phase 1). The duplicate count in lane 4 can be C0 from lane 0, C4 from lane 4, or C4+C0. In phase 2, when lane {2,4} is merged, the possible output combinations are:

[0113] [Table 5]

[0114] An exemplary pseudocode implementation for performing duplicate counting in a 4×4 odd-odd / even-even merge stage in phase 1 is shown in Table 5:

[0115] [Table 6] TIFF0007640752000007.tif35170

[0116] The implementation is similar to the 2x2 merge, e.g., as shown in Table 4, but with the value represented by out_incr latched to the Cm value described above. The out_incr value is used to modify the final overlap count of the 4x4 odd-odd / even-even merge stage in phase 2.

[0117] An exemplary pseudocode implementation for performing duplicate counting in a 4×4 odd-odd / even-even merge stage is shown below in Phase 2 and in Table 6:

[0118] [Table 7] TIFF0007640752000009.tif90170

[0119] In lines 3-13, the larger increment value incr_large is set to the increment value of the larger lane, and the smaller increment value incr_small is set to the increment value of the smaller lane. In lines 15-18, the count count_large of the larger lane is adjusted to compensate for the potential added duplicate value from the previous stage 1.

[0120] In stage 6, a 4x4 odd-even merge. Similar to the 2x2 odd-even merge in stage 3, the i-th and i+1-th elements are compared and merged. The output is a single sorted list of length 8.

[0121] After stage 6, the value shuffle network shuffles values ​​based on the compositing operations performed on the configured XPUs. For vector sort operations, the value shuffle network in source lane mode routes values ​​from their original lane to the appropriate destination lane based on where the corresponding sorted keys ended up after sorting. For duplicate count operations, the value shuffle network in destination lane mode routes each lane's counts from the original lane to the appropriate destination lane based on where the corresponding counted values ​​ended up after sorting.

[0122] In some examples, the XPUs may be configured to perform a floating point MIN / MAX scan to scan the input for a minimum or maximum value, respectively. The XPUs may also be configured to scan the indices of received tuples across the processing lanes, for example to identify a maximum or minimum index. As part of performing a composition operation to scan operation, the processing cells may be configured to perform a comparison between floating point values, e.g., 32-bit floating point values, or between unsigned integers. Primitive operations for comparison may be used for other composition operations, such as sorting or counting duplicate values, as described herein.

[0123] 8B-8G show each stage of the merge sort network 800A on an exemplary input. For purposes of illustration, the exemplary keys sorted are: Key = {5, 6, 7, 6, 5, 5, 5, 5} Lane Identifier={0, 1, 2, 3, 4, 5, 6, 7} The lane identifiers for the lanes streaming each respective key are provided for reference.

[0124] FIG. 8B illustrates an exemplary stage 1 801A of an XPU odd-even merge network 800. In stage 1 801A, keys from adjacent data processing cells are processed by the respective MIN and MAX cells. For example, in stage 1, the MIN cell in lane 0 determines the minimum value in lanes 0 and 1, and the MAX cell in lane 1 determines the maximum value of the keys in lanes 0 and 1. The crossbar in stage 1 is configured to provide the keys in lane 1 to the MIN cell and the keys in lane 0 to the MAX cell. The output of stage 1 is four sorted lists of keys, each of length 2. According to the example, lanes 2 and 3 have their keys swapped. The keys for other pairs of lanes, i.e., lanes 0 and 1, lanes 4 and 5, and lanes 6 and 7, pass through stage 1 without swapping, since the respective keys for that lane are already sorted. The counts in lanes 5 and 7 are incremented (the higher of each compared lane with the same key).

[0125] FIG. 8C illustrates an example stage 2 800C of an odd-even merge network of an XPU according to an embodiment of the disclosure. In the second stage, adjacent lists are merged and keys in odd positions of the sorted list from stage 1 are compared. Keys in even positions of the sorted list are similarly compared. In stage 2, for example, the processing cells in lanes 0 and 1 are MIN cells, while the processing cells in lanes 2 and 3 are MAX cells. As the higher lanes of the compared pair (4,6)(5,7) all have key 5, the duplicate counts for lanes 6 and 7 are incremented.

[0126] 8D illustrates an example stage 3 of an odd-even merge network of an XPU according to an embodiment of the disclosure. In the third stage, the remaining keys in the middle lanes of each sorted list, i.e., lanes (1,3) and (5,6), are compared. The duplicate count for lane 3 is incremented because 6=6, and the duplicate count for lane 6 is incremented because 5=5.

[0127] FIG. 8E illustrates an example stage 4 of an odd-even merge network of an XPU according to an embodiment of the disclosure. The fourth stage is phase 1 of a 4×4 even-even / odd-odd merge. Lanes 0,4; 1,5; 2,6; and 3,7 are compared and swapped as necessary. The duplicate count for lane 4 is incremented (lane 1 had a value of 5 and lane 4 had a value of 5). The increment value for lane 4 is also set to 1 (larger lane 4 count). In this example, all compared pairs are swapped.

[0128] 8F illustrates an example stage 5 of an odd-even merge network of an XPU according to an embodiment of the disclosure. The values ​​from lanes 6 and 4 are compared (and swapped), and the values ​​from lanes 7 and 1 are also compared (and swapped). Since lanes 4 and 6 belong to the same half of the sorted list, the count in lane 6 is increased by 1.

[0129] 8G illustrates an example stage 6 of an odd-even merge network of an XPU, according to an embodiment of the disclosure. Values ​​from lanes 5 and 4 are compared and swapped. Values ​​from lanes 6 and 7 are compared, and values ​​from lanes 1 and 3 are compared. Counts for lanes 4, 7, and 3 are incremented.

[0130] 8H illustrates an example value shuffle network for an odd-even merge network of an XPU, according to an aspect of the disclosure. After shuffling, the counts are output in ascending order: {1, 1, 1, 2, 2, 3, 4, 5}.

[0131] Other Exemplary Compositing Operations In addition to vector sort and duplicate count, the XPU can be configured with a processing network of cells and crossbars to perform a variety of other operations described herein, including, for example, vector partition, vector histogram, vector replace, vector shift insert, vector gather, and vector scatter.

[0132] Vector Partition The XPU can be configured to partition an input vector by element according to a range of values ​​that defines a number of bins. The XPU partitions an input element into the smallest bin whose value is greater than or equal to the input element. The XPU can perform vector partitioning, e.g., as described herein, as part of a compositing operation to convert an input sparse matrix from one format to another, such as from compressed row storage (CRS) to coordinate list (COO) format.

[0133] In each data processing lane, a respective processing cell receives an input value, such as an unsigned integer, to be partitioned. The input value can be an element of a vector or other data structure. The processing cell also receives a tuple of a bin value and a mask value for the input value. The input value and each of the elements of the bin tuple are the same data type, such as a 32-bit unsigned integer. The bin tuple is indexed with the first element of the tuple having an index of zero in an XPU implementation using zero indexing, or 1 otherwise. The second bin element has an index of 1 (or 2), the third bin element has an index of 2 (or 3), and so on.

[0134] For a bin tuple, bin[y] is the element of the tuple at index y. In some examples, the mask value is a mask for the input value. In other examples, the mask bits is instead a tuple of one or more mask bits, where the index of each bit in the mask tuple corresponds to a respective bin element in the input bin tuple. For example, if an element mask[i] is equal to zero, then the bin element bin[i] at the same index i is masked and not processed as an input by the XPU.

[0135] The output to a vector partition is a partition identifier, which is the index of the bin value into which the input value is partitioned. The partition identifier is based on the range defined by the tuple of bin values. The partition index can have values ​​from zero to one less than the number of data processing lanes for the XPU (or from one to the number of data processing lanes, if the partition identifier is one-indexed instead of zero-indexed).

[0136] For a given element values[i] at index i of the input vector, the partition identifier is the lowest value k such that bin element bin[k] is less than or equal to the elements of the input vector at values[i], which is less than bin element bin[k+1]. In these examples, the last bin element of the bin tuple is set to the maximum value of the data type for the input values ​​and bin elements. For example, if the XPU is performing vector partitioning on a vector of 32-bit unsigned integer elements, the last bin element of the bin tuple is set to INT_MAX, which represents the highest value for a 32-bit unsigned integer.

[0137] If an element in values[i] is not in the range of a bin element, the resulting partition identifier is masked when sent as output by the XPU. Also, if a bin tuple does not contain monotonically increasing integer elements, the partition identifier for the input element is undefined. In other words, a bin tuple must contain a strictly increasing sequence of elements.

[0138] An exemplary set of inputs to the XPU may be as follows: Value = {5, 7, 10, 1, 512, 8, 1023, 0} mask={0, 1, 0, 0, 1, 1, 1, 1} bins = {0, 7, 256, 1024, 1, 0, 0, 1} In this example, the mask values ​​correspond to input values, not bin values. Each lane of an 8-lane XPU can receive a respective input element from, for example, an input vector, and its corresponding mask bit. The output partition identifier is: Partition identifiers: {0, 1, 1, 0, 2, 1, 2, 0} It is.

[0139] Based on the mask value, only the input values ​​7, 512, 8, and 1023 are partitioned. Furthermore, there are four bins specified in the bin tuple because 0, 7, 256, and 1024 are the only strictly increasing sequences of integer values. The masked value has a partition identifier of zero. Of the remaining input values, 7 is mapped to partition identifier 1 (because the first bin, 0, is less than or equal to 7, but 7 is not less than 7). In these examples, bin[-1] is defined as the minimum negative value of the data type for the input value and bin element. For example, the first bin element of the bin tuple is set to -INT_MAX, which represents the lowest value of a 32-bit unsigned integer multiplied by negative 1 when the XPU performs vector partitioning on a vector of 32-bit unsigned integer elements.

[0140] In some examples, the rule applied by the XPU in determining the partition identifier for a value in value[i] is the lowest value k such that bin element bin[k-1] is less than values[i] and values[i] is less than or equal to bin[k].

[0141] Vector Histogram The XPU can be configured to count the number of occurrences of an input value in a vector. The output by the XPU is a list of unique elements along with a count of their occurrences. The input to the XPU also contains a mask bit for each input value that counts only the occurrences of the unmasked value.

[0142] Exemplary inputs to the XPU can be as follows: Value = {0, 0, 1, 0, 2, 1, 2, 100} mask={0, 0, 0, 0, 0, 0, 0, 1} In this example, the value 100 is masked and is not considered by the XPU in vector histogram operations.

[0143] An example output by the XPU may be as follows: Value = {0, 1, 2, 0, 0, 0, 0, 0} Count = {3, 2, 2, 0, 0, 0, 0, 0} mask = {0, 0, 0, 1, 1, 1, 1, 1} The values ​​are output first by the eigenvalues, i.e., 0, 1, and 2. Each element in the list of count values ​​corresponds to a count of each eigenvalue, i.e., three zeros, two ones, and two twos in the input. Finally, the XPU also outputs masks for the remaining output vector elements that were previously masked or do not contain eigenvalues ​​in the input. The input vector is pre-sorted, so that each element in the list of count values ​​can be matched with a corresponding eigenvalue from smallest to largest (or vice versa, depending on the sorting).

[0144] Vector Compact The XPU can be configured to remove masked elements of an input vector starting at the input offset and return an output vector of the removed elements. In each data processing lane, a respective processing cell receives an input value and a mask bit for that value, as well as an offset. If the input value is not masked, the input value is bypassed by the processing cell.

[0145] An exemplary set of inputs to the XPU may be as follows: Value = {0, 1, 2, 3, 4, 5, 6, 7} mask = {1, 0, 1, 0, 1, 0, 0, 1} Offset=5 From the received values, values ​​0, 2, 4, 7 are masked out for removal. The offset is 5, so an example output by the XPU could be: Values ​​= {7, 1, 3, 5, 6, 0, 2, 4} mask={0, 1, 1, 1, 1, 0, 0, 0} Because the offset is 5, but the fifth element in the input value is not masked, the XPU continues through the input values ​​to the first masked value, 7. The XPU then contains in the output vector the previously unmasked values ​​of the input vector; these values ​​are now masked. The remaining values ​​in the output vector are the masked values ​​of the input vector in the order of their appearance in the input vector.

[0146] Vector Reduction The XPU can be configured to reduce an input vector, such as by summing each element of the input vector and returning the sum or returning the maximum or minimum element in the input vector. For example, the XPU can reduce an input vector by summing each element of the vector. For vectors larger than the total number of data processing lanes, the XPU can maintain a running total for summing elements of each segment of the input vector up to the size of the total number of lanes.

[0147] Vector Replacement The XPU can be configured to permute input vectors according to a provided pattern. The input vectors can be reshaped, for example to be provided as input to a neural network layer that requires inputs of a predefined shape. The XPU can permute vectors as large as the number of data processing lanes for the XPU. Vectors larger than the total number of data processing lanes can be permuted into smaller segments, up to the total number of lanes.

[0148] Vector Shift Insert The XPU can be configured to receive two vectors as operands, called v0 and v1, and a scalar offset value. The scalar offset defines the starting point in a third vector, v3, if vectors v0 and v1 are combined. The output is a selection of vectors as long as the number of data processing elements in the XPU, starting at the first value at the offset in vector v3. The offset can be an integer value between zero and one less than the number of data processing elements.

[0149] An exemplary set of inputs to the XPU may be as follows: Vector v0={1, 2, 3, 4, 5, 6, 7, 8} Vector v1={a, b, c, d, e, f, g, h} Scalar offset value = 6 In this example, vector v1 contains characters to illustrate how the XPU can generate an output vector from both input vectors to a vector shift insert. The selected vector from performing the vector shift insert is: Selected Vector = {7, 8, a, b, c, d, e, f} It is.

[0150] Since the first element in vector v0 at offset 6 is 7, the first two elements of the selected vector are 7 and 8. Then the next seven elements are selected, ending in vector v0 and continuing through vector v1.

[0151] 9 is a flow chart of an example process 900 for performing compositing operations in an XPU. A hardware circuit can be configured with an XPU that includes multiple stages. Each stage can include a crossbar and two or more processing cells. Multiple data processing lanes stream respective data from an upstream input to a downstream destination through the cells and crossbar of that stage.

[0152] According to block 910, the hardware receives input data from an upstream input along a data processing lane and a first instruction to perform a first operation. The upstream input can be a processor or a memory device from which data is sent for processing in the XPU of the hardware circuit. The first operation is a compositing operation, such as vector scan, vector segmentation scan, vector sort, vector duplicate count, etc. The first instruction can be a control signal representing the compositing operation.

[0153] According to block 920, in response to receiving the first instruction, for each stage, the hardware circuitry is configured to transmit a second instruction to a respective processing cell of that stage, each cell being configured to perform a respective second operation in response to receiving an input from a respective data processing lane. The second operation is a primitive operation performed by the processing cell. As described herein with reference to Figures 3 and 4, the instruction may be received from a respective control cell corresponding to each processing cell.

[0154] According to block 930, in response to receiving the first instruction, for each stage, the hardware circuitry transmits a respective third instruction to a respective crossbar for each stage.

[0155] According to block 940, the hardware circuitry performs the first operation by processing the input data received along a plurality of data processing lanes and a plurality of cells configured to perform respective second operations. In configurations 500-600, such as those shown and described with reference to Figures 5 and 6, the cells and crossbars form a processing network for performing the first operation.

[0156] FIG. 10 is a flow chart of an example process 1000 for sorting and generating duplicate counts for input vectors to an XPU in accordance with an aspect of the disclosure.

[0157] According to block 1010, the hardware circuitry configures the cross-lane processing units to form a processing network of processing cells and crossbars. For example, the hardware circuitry may receive instructions as described herein with reference to FIG. 9 to generate a vector sort and duplicate count processing network, as described herein with reference to FIG. 8A-8H.

[0158] According to block 1020, a hardware circuit receives an input vector of elements. According to block 1030, the hardware circuit generates a sorted output vector and data specifying a count of duplicate elements in the input vector. Exemplary generation of the sorted output vector and data specifying the duplicate count is described herein with reference to Figures 8A-8H.

[0159] Exemplary Computing Environment 11 is a block diagram of an exemplary environment 1100 for executing the hardware circuit 101. The hardware circuit 101 can be executed in a device having one or more processors in one or more locations, such as a server computing device 1115. The user computing device 1112 and the server computing device 1115 can be communicatively coupled to one or more storage devices 1130 over a network 1160. The storage device 1130 can be a combination of volatile and non-volatile memory and can be in the same or different physical location as the computing devices 1112, 1115. For example, the storage device 1130 can include any type of non-transitory computer-readable medium that can store information, such as a hard drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, writable, and read-only memory.

[0160] The server computing device 1115 may include one or more processors 1113 and memory 1114. The memory 1114 may store information accessible by the processor 1113, including instructions 1121 that may be executed by the processor 1113. The memory 1114 may also include data 1123 that may be retrieved, manipulated, or stored by the processor 1113. The memory 1114 may be one type of non-transitory computer-readable medium that may store information accessible by the processor 1113, such as volatile and non-volatile memory. The processor 1113 may include one or more central processing units (CPUs), graphic processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs). The processor 1113 may include a co-processor executed as part of a hardware circuit, as described herein with reference to FIG. 1.

[0161] The instructions 1121 may include one or more instructions that, when executed by the processor(s) 1113, cause the one or more processors to perform an action defined by the instructions. The instructions 1121 may be stored in object code format for direct processing by the processor(s) 1113, or in other formats, including interpretable scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. The instructions 1121 may include instructions for configuring an XPU of the hardware circuit 101 consistent with aspects of the disclosure. The server computing device 1115 and / or the user computing device 1112 may execute a compiler or other program to generate and send instructions to the hardware circuit 101 as control signals to configure the XPU of the circuit.

[0162] The data 1123 may be retrieved, stored or modified by the processor 1113 according to the instructions 1121. The data 1123 may be stored in a computer register, in a relational or non-relational database, as a table with multiple different fields and records, or as a JSON, YAML, Proto or XML document. The data 1123 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII or Unicode. Additionally, the data 1123 may include sufficient information to identify the associated information, such as numbers, descriptive text, proprietary code, pointers, references to data stored in other memory, including other network locations, information used by a function to calculate the associated data, etc.

[0163] The user computing device 1112 may be configured similarly to the server computing device 115, having one or more processors 1116, memory 1117, instructions 1118, and data 1119. The user computing device 1112 may also include a user output 1126 and a user input 1124. The user input 1124 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.

[0164] The server computing device 1115 may be configured to transmit data to the user computing device 1112, which may be configured to display at least a portion of the received data on a display running as part of the user output 1126. The user output 1126 may also be used to display an interface between the user computing device 1112 and the server computing device 1115. The user output 1126 may alternatively or additionally include one or more speakers, transducers or other audio output, a haptic interface, or other haptic feedback that provides non-visual and non-auditory information to a platform user of the user computing device 1112.

[0165] Although FIG. 11 illustrates the processors 1113, 1116 and memories 1114, 1117 as being within the computing devices 1115, 1112, the components described herein, including the processors 1113, 1116 and memories 1114, 1117, may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, some of the instructions 1121, 1118 and data 1123, 1119 may be stored on a removable SD, in a read-only computer chip, and so forth. Some or all of the instructions and data may be stored in locations physically separate from, but still accessible to, the processors 1113, 1116. Similarly, the processors 1113, 1116 may include a collection of processors that may perform simultaneous and / or sequential operations. The computing devices 1115, 1112 may each include one or more internal clocks that provide timing information that may be used for time measurements for operations and programs executed by the computing devices 1115, 1112.

[0166] The server computing device 1115 can be configured to receive requests to process data from the user computing device 1112. For example, the environment 1100 can be part of a computing platform configured to provide various services to a user via various user interfaces and / or APIs that expose platform services. One or more services can be a machine learning framework or a set of tools for generating a neural network or other machine learning model according to a specified task and training data. The user computing device 1112 can receive and send data specifying a type of workload or synthesis operation that the XPU of the hardware circuit 101 is to be configured to perform. The user computing device 1112 can send instructions directly to the hardware circuit 101 or have the server computing device 1115 generate and send instructions as control signals to the hardware circuit 101 as described herein.

[0167] The devices 1112, 1115 can communicate directly or indirectly over the network 1160. The devices 1115, 1112 can set up listening sockets that may accept initiating connections to send and receive information. The network 1160 itself can include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks using one or more corporate proprietary communication protocols. The network 1160 can support a variety of short-range and long-range connections. The short-range and long-range connections can be made over different bandwidths, such as 2.402 GHz to 2.480 GHz (commonly associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (commonly associated with the Wi-Fi® communication protocol), or by various communication standards, such as the LTE® standard for wireless broadband communication. The network 1160 can additionally or alternatively support wired connections between the devices 1112, 1115, including over various types of Ethernet connections.

[0168] Although a single server computing device 1115 and user computing device 1112 are shown in Figure 11, it is understood that aspects of the disclosure can be performed according to a variety of different configurations and quantities of computing devices, including in paradigms for sequential or parallel processing, or on a distributed network of multiple devices. In some implementations, aspects of the disclosure can be performed on a single device or any combination thereof.

[0169] In view of the foregoing, aspects of the disclosure are directed to a cross-lane processing unit (XPU) for performing data-dependent operations across multiple data processing lanes of a processor. Rather than executing operation-specific circuitry for each data-dependent operation, the XPU can be configured to perform different operations in response to input signals that constitute the individual operations performed by processing cells and crossbars arranged as stacked networks in the XPU. Each processing cell can receive and process data across multiple data processing lanes. Aspects of the disclosure include configuring the XPU to use a vector sort network to perform duplicate counting, eliminating the need to configure the XPU separately for sorting and duplicate counting.

[0170] Aspects of the present disclosure may be implemented in digital circuitry, in a computer-readable storage medium, as one or more computer programs, or as a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example as one or more instructions executable by a cloud computing platform and stored on a tangible storage device.

[0171] In this specification, the term "configured to" is used in different contexts relating to a computer system, hardware and hardware circuits, or parts of a computer program, engine, or module. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed on the system that, when operated, causes the system to perform one or more operations. When a piece of hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, when operated, receive inputs and generate outputs corresponding to the one or more operations according to the inputs. When a computer program, engine, or module is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0172] Although the operations illustrated in the figures and recited in the claims are shown in a particular order, it is understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed more than once, and / or performed in parallel with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring the components to be separated. The described components, modules, programs, and engines may be integrated as a single system or may be part of multiple systems.

[0173] Unless otherwise stated, the above alternative examples are not mutually exclusive and may be implemented in various combinations to achieve unique advantages. These and other variations and combinations of the features described above can be utilized without departing from the subject matter defined by the claims, and therefore the above description of the examples should be construed as illustrative, rather than limiting, the subject matter defined by the claims. In addition, the provision of examples described herein, as well as phrases such as "including," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example. Rather, the examples are intended to illustrate only one of many possible implementations. Furthermore, the same reference numbers in different drawings may identify the same or similar elements.

[0174] With respect to the use of substantially any plural and / or singular terminology herein, e.g., "an / that element," "one or more elements," "multiple elements," "plural elements," "at least one element," etc. (where the term "element" is in place of any system, component, data, etc.), those of skill in the art can convert from plural to singular and / or from singular to plural as appropriate for the described context and / or application. The various singular / plural permutations may be expressly set forth herein for clarity and without limitation unless expressly stated otherwise.

Claims

1. A hardware circuit comprising: a plurality of stages, each stage including a crossbar and two or more cells; a plurality of data processing lanes for streaming respective data from an upstream input to a downstream destination through the plurality of cells and the plurality of crossbars of the plurality of stages; The hardware circuit includes: receiving input data from the upstream input along the plurality of data processing lanes and receiving a first instruction to perform a first operation; in response to receiving the first instruction, for each stage: sending respective second instructions to respective processing cells of said stage, each cell being configured to perform a respective second operation in response to receiving an input from a respective data processing lane; configured to transmit respective third instructions to respective crossbars for the stages, the crossbars configured to permute outputs from each cell of the stage to a cell of a subsequent stage along the plurality of data processing lanes; a hardware circuit configured to perform the first operation by processing the input data received along the plurality of data processing lanes and the plurality of cells configured to perform respective second operations.

2. 2. The hardware circuit of claim 1, wherein each cell is configured to receive a respective first input operand from a respective data processing lane passing through the cell and a respective second input operand from a respective crossbar of a stage upstream of the cell.

3. 2. The hardware circuit of claim 1 , wherein the downstream destination of data of the multiple data processing lanes is a vector processing unit, the vector processing unit configured to perform single instruction multiple data vector operations on output data of the hardware circuit.

4. each of the cells is configured to perform one or more of a plurality of predefined primitive operations in response to one or more received instructions; the hardware circuit further comprises a plurality of control cells; 2. The hardware circuit of claim 1, wherein upon transmitting the respective second instructions to the respective processing cells, the hardware circuit is configured to generate and transmit, by each control cell, a respective control signal to each processing cell based on the first operation specified by the first instruction.

5. 5. The hardware circuit of claim 4, wherein in generating and transmitting the respective control signals by each control cell, the hardware circuit is configured to generate a respective control signal to cause each processing cell to perform one of a respective arithmetic operation, a comparison operation, and a bypass operation based on at least one of the stage in which the processing cell resides or the data processing lane passing through the processing cell.

6. 5. The hardware circuit of claim 4, wherein the plurality of cells and plurality of crossbars form a processing network of connected cells across the plurality of stages and plurality of data processing lanes, the processing network of connected cells configured to receive the input data and generate respective output data in accordance with performing the first operation on the input data.

7. The processing network of connected cells is configured to perform a combined vector sort and duplicate count operation, the combined vector sort and duplicate count operation comprising: receiving an input vector of elements by said processing network; and generating, by said processing network, as output, data specifying a sorted output vector and a count of duplicate elements in said input vector.

8. 2. The hardware circuit of claim 1, wherein the input data comprises sparse vector data, and after transmitting the respective second instruction and the respective third instruction, the hardware circuit is configured to perform one of a vector scan, a vector sum, a vector sort, or a vector duplicate count.

9. 1. A system comprising: each stage comprising a hardware circuit including a plurality of stages including a crossbar and two or more cells, and a plurality of data processing lanes for streaming respective data from an upstream input to a downstream destination via the plurality of cells of the plurality of stages and the plurality of crossbars; The hardware circuit includes: receiving input data from the upstream input along the plurality of data processing lanes and receiving a first instruction to perform a first operation; in response to receiving the first instruction, for each stage: sending respective second instructions to respective processing cells of said stage, each cell being configured to perform a respective second operation in response to receiving an input from a respective data processing lane; configured to transmit respective third instructions to respective crossbars for the stages, the crossbars configured to permute outputs from each cell of the stage to a cell of a subsequent stage along the plurality of data processing lanes; a system configured to perform the first operation by processing the input data received along the plurality of data processing lanes and the plurality of cells configured to perform respective second operations.

10. 10. The system of claim 9, wherein each cell is configured to receive a respective first input operand from a respective data processing lane passing through the cell and a respective second input operand from a respective crossbar of an upstream stage of the cell.

11. 10. The system of claim 9, wherein the downstream destination of the data of the multiple data processing lanes is a vector processing unit, the vector processing unit configured to perform single instruction multiple data vector operations on output data of the hardware circuit.

12. each of the cells is configured to perform one or more of a plurality of predefined primitive operations in response to one or more received instructions; the hardware circuit further comprises a plurality of control cells; 10. The system of claim 9, wherein upon transmitting the respective second instructions to the respective processing cells, the hardware circuitry is configured to generate and transmit, by each control cell, a respective control signal to each processing cell based on the first operation specified by the first instruction.

13. 13. The system of claim 12, wherein in generating and transmitting the respective control signals by each control cell, the hardware circuitry is configured to generate a respective control signal to cause each processing cell to perform one of a respective arithmetic operation, a comparison operation, and a bypass operation based on at least one of the stage in which the processing cell resides or the data processing lane passing through the processing cell.

14. 13. The system of claim 12, wherein the plurality of cells and plurality of crossbars form a processing network of connected cells across the plurality of stages and plurality of data processing lanes, the processing network of connected cells configured to receive the input data and generate respective output data in accordance with performing the first operation on the input data.

15. The processing network of connected cells is configured to perform a combined vector sort and duplicate count operation, the combined vector sort and duplicate count operation comprising: receiving an input vector of elements by said processing network; and generating, by said processing network, as output, data specifying a sorted output vector and a count of duplicate elements in said input vector.

16. 10. The system of claim 9, wherein the input data comprises sparse vector data, and after transmitting the respective second instruction and the respective third instruction, the hardware circuitry is configured to perform one of a vector scan, a vector sum, a vector sort, or a vector duplicate count.

17. 1. A computer-implemented method comprising: a hardware circuit comprising a plurality of stages, each stage including a crossbar and two or more cells, and a plurality of data processing lanes for streaming respective data from an upstream input to a downstream destination via a plurality of cells of the plurality of stages and a plurality of crossbars, the hardware circuit receiving input data from the upstream input along the plurality of data processing lanes and a first instruction for performing a first operation; in response to receiving the first instruction, for each stage: the hardware circuitry is configured to send a respective second instruction to a respective processing cell of the stage, each cell being configured to perform a respective second operation in response to receiving an input from a respective data processing lane; sending, by the hardware circuitry, a respective third instruction to a respective crossbar for the stage, the crossbar being configured to permute outputs from each cell of the stage to a cell of a next stage along the plurality of data processing lanes; performing, by the hardware circuitry, a first operation by processing the input data received along the plurality of data processing lanes and the plurality of cells configured to perform respective second operations.

18. 20. The method of claim 17, wherein each cell is configured to receive a respective first input operand from a respective data processing lane passing through the cell and a respective second input operand from a respective crossbar of an upstream stage of the cell.

19. 20. The method of claim 17, wherein the downstream destination of data of the multiple data processing lanes is a vector processing unit, the vector processing unit configured to perform single instruction multiple data vector operations on output data of the hardware circuit.

20. each of the cells is configured to perform one or more of a plurality of predefined primitive operations in response to one or more received instructions; the hardware circuit further comprises a plurality of control cells; 20. The method of claim 17, wherein transmitting the respective second instructions to the respective processing cells comprises each control cell generating and transmitting a respective control signal to each processing cell based on the first operation specified by the first instruction.

Images