271results about "Systolic arrays" patented technology

The invention discloses an accelerator for accelerating the reasoning process of an unstructured sparse large language model and an operation method of the accelerator. The accelerator comprises a processor module, a direct memory access module, a memory controller module, a DDR memory module, an interconnection bus module, a calculation core module and a router module. The interconnection bus module is connected with the direct memory access module, the memory controller module and the computing core module; the processor module is connected with the direct memory access module; the DDR memory module is connected with the memory controller module; the router module is connected with the computing core module and the adjacent router module; and the calculation core module obtains to-be-calculated data from the off-chip DDR memory module through the interconnection bus module. According to the method, the limit that data reuse is carried out only along a single row or a single column on a traditional systolic array is broken through, cross-row and cross-column data reuse is achieved, the pressure of hardware equipment is relieved, the positions of non-zero elements are coded through bit masks, and the calculation efficiency is improved.

Systems and methods are provided to enable parallelized multiply-accumulate operations in a systolic array. Each column of the systolic array can include multiple busses enabling independent transmission of input partial sums along the respective bus. Each processing element of a given columnar bus can receive an input partial sum from a prior element of the given columnar bus, and perform arithmetic operations on the input partial sum. Each processing element can generate an output partial sum based on the arithmetic operations, provide the output partial sum to a next processing element of the given columnar bus, without the output partial sum being processed by a processing element of the column located between the two processing elements that uses a different columnar bus. Use of columnar busses can enable parallelization to increase speed or enable increased latency at individual processing elements.

An apparatus for accelerating neural networks, includes: a memory for storing graph input data including vertices and edges; an aggregation engine that processes the accumulation of features and generates feature vectors by taking the graph input data and performing an aggregation operation on the graph input data; an on-chip cache for caching the feature vectors; and a combination engine that generates a systolic array for matrix multiplications based on the feature vectors taken from the on-chip cache and weights taken from the memory.

An apparatus for facilitating support of 8-bit floating point format operands in a computing architecture is disclosed. In one embodiment, the apparatus includes a processor comprising: a decoder to decode an instruction fetched for execution into a decoded instruction, where the decoded instruction is a matrix instruction that operates on an 8-bit floating point operand to cause the processor to perform a parallel dot product operation; a controller to schedule the decoded instruction and provide input data for an 8-bit floating point operand according to an 8-bit floating point data format indicated by the decoded instruction; and a pulsation dot product circuit to execute the decoded instruction using pulsation layers, each pulsation layer comprising one or more sets of interconnected multipliers, shifters and adders, each set of multipliers, shifters and adders to generate a dot product of 8-bit floating point operands.

Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and / or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Disclosed is a method and apparatus with deep learning operations. A deep learning apparatus includes a processor, configured to support a plurality of different operation modes, including a systolic array having a plurality of multiplier accumulator (MAC) units, and a control circuit configured to respectively control, for each the plurality of different operation modes, select operations of the plurality of MAC units and data movements among the plurality of MAC units.

A disclosed approximate tensor processing unit (APTPU) includes two main components: (1) approximate processing elements (APEs) consisting of a low-precision multiplier and an approximate adder, and (2) pre-approximate units (PAUs) which are shared among the APEs in the APTPU's systolic array, functioning as the steering logic to pre-process the operands and feed them to the APEs. Performance of the disclosed APTPU across various configurations and various workloads shows that the disclosed APTPU's systolic array achieves delay, area, and power reductions, while realizing comparable accuracy to previous designs.

One embodiment provides a graphics processor comprising a base die including a plurality of chiplet sockets and a plurality of chiplets coupled with the plurality of chiplet sockets. At least one of the plurality of chiplets including a plurality of processing elements, a distributed shared local memory coupled with the plurality of processing elements, a plurality of matrix engines coupled with the distributed shared local memory, and an asynchronous matrix multiply accumulate (MMA) controller configured to perform an asynchronous MMA operation via the plurality of matrix engines.

Methods and apparatus relating to techniques for multi-tile memory management. In an example, a graphics processor includes an interposer, a first chiplet coupled with the interposer, the first chiplet including a graphics processing resource and an interconnect network coupled with the graphics processing resource, cache circuitry coupled with the graphics processing resource via the interconnect network, and a second chiplet coupled with the first chiplet via the interposer, the second chiplet including a memory-side cache and a memory controller coupled with the memory-side cache. The memory controller is configured to enable access to a high-bandwidth memory (HBM) device, the memory-side cache is configured to cache data associated with a memory access performed via the memory controller, and the cache circuitry is logically positioned between the graphics processing resource and a chiplet interface.

Systems and methods for improving cache efficiency and utilization are disclosed. In one embodiment, a graphics processor includes processing resources to perform graphics operations and a cache controller of a cache coupled to the processing resources. The cache controller is configured to control cache priority by determining whether default settings or an instruction will control cache operations for the cache.

An apparatus comprises decode circuitry to decode a single matrix instruction having fields to indicate an opcode and locations of a first source matrix including a first plurality of 8-bit floating point data elements encoded in a first 8-bit floating point format, a second source matrix including a second plurality of 8-bit floating point data elements encoded in a second 8-bit floating point format, and a third source matrix including a plurality of 32-bit floating point data elements. The apparatus further comprises execution circuitry, responsive to the single matrix instruction, to generate a plurality of products based on the first plurality of 8-bit floating point data elements of the first source matrix and the second plurality of 8-bit floating point data elements of the second source matrix, and accumulate each product of the plurality of products with a corresponding 32-bit floating point data element of the third source matrix to generate a corresponding 32-bit floating point result data element of a result matrix.

A hardware accelerator including tiles arranged in a systolic array. At each of the tiles, the systolic array receives a first input block that includes first input matrix elements of a first input matrix. In each of a plurality of multiplication iterations, at each of the tiles, the systolic array receives a respective second input block. The systolic array computes tile products of the first input matrix elements and second input matrix elements included in the second input blocks. The systolic array adds the tile products to column-wise partial sums and transmits the column-wise partial sums to subsequent tiles along accumulator rings included in array columns of the systolic array. In a subset of the multiplication iterations, the systolic array outputs product block rows of a product matrix. The product block rows each include product matrix blocks computed as rows of the column-wise partial sums.

A processing apparatus includes a processing resource including a general-purpose parallel processing engine and a matrix accelerator. The matrix accelerator includes first circuitry to receive a command to perform operations associated with an instruction, second circuitry to configure the matrix accelerator according to a physical depth of a systolic array within the matrix accelerator and a logical depth associated with the instruction, third circuitry to read operands for the instruction from a register file associated with the systolic array, fourth circuitry to perform operations for the instruction via one or more passes through one or more physical pipeline stages of the systolic array based on a configuration performed by the second circuitry, and fifth circuitry to write output of the operations to the register file associated with the systolic array.

Embodiments are generally directed to graphics processor data access and sharing. An embodiment of an apparatus includes a circuit element to produce a result in processing of an application; a load-store unit to receive the result and generate pre-fetch information for a cache utilizing the result; and a prefetch generator to produce prefetch addresses based at least in part on the pre-fetch information; wherein the load-store unit is to receive software assistance for prefetching, and wherein generation of the pre-fetch information is based at least in part on the software assistance.

Described is a device having three-dimensional (3D) integration of a DRAM die and processor ASIC. Such an integrated device achieves much higher memory bandwidth and low memory access power consumption than prior art devices while still retaining the high capacity and low cost of DRAMs. The devices and techniques described herein can result in much higher computational throughput, higher power efficiency, and lower cost per throughput than conventional processors while still providing the high memory capacity necessary to be able to work on large problems.

Aspects of the disclosure provide for methods, systems, and apparatuses, including computer-readable storage media, for sparse matrix multiplication. A system for matrix multiplication includes an array of sparse shards. Each sparse shard can be configured to receive an input sub-matrix and an input sub-vector, where the input sub-matrix has a number of non-zero values equal to or less than a predetermined maximum non-zero threshold. The sparse shard can, by a plurality of multiplier circuits, compute one or more products of vector values multiplied with respective non-zero values of the input sub-matrix. The sparse shard can generate, as output to the sparse shard and using the one or more products, a shard output vector that is the product of applying the shard input vector to the shard input matrix.

A quantization method of a deep learning neural network model is disclosed. An embodiment of the disclosure provides a quantization method comprising: detecting a feature change of input data caused by a change in an external environment, from input image data of a quantized deep learning neural network model based on a plurality of preset quantization parameters; performing quantization calibration for the deep learning neural network model to determine a new quantization parameter corresponding to the feature change of input data caused by the change in the external environment; and updating at least one of the plurality of preset quantization parameters based on the new quantization parameter.

A compiler tool and a method of operating a compiler tool for selecting between executing a matrix multiplication operation in a weight stationary mode or in an output stationary mode on a systolic array with reconfigurable processing elements are presented. The compiler tool receives configuration parameters, energy parameters, and performance parameters as well as dimensions of the matrices to be multiplied and estimates energy consumptions and performance numbers for executing the matrix multiplication operation in the weight stationary and the output stationary modes. The compiler tool selects between operating the matrix multiplication operation in the weight stationary and the output stationary mode based on the estimated energy consumption and the estimated performance numbers.

A system and method for matrix multiplication using a systolic array configurable between a plurality of operating modes. A pulsation processor may receive a data type indicator for matrix multiplication. For a first data type, the systolic processor may load right-hand side data from a right-hand matrix register into data processing units of the systolic array between rows 0 and M-1, and pass respective rows of left-hand side data through corresponding rows of the systolic array between rows 0 and M-1. For the second data type, the pulsation processor may segment each element of the left-hand-side data and the right-hand-side data into a respective first element half and a respective second element half, and move each element half through a corresponding row of the pulsation array between rows 0 and 2M-1.

Embodiments described herein provided for an instruction and associated logic to enable a processing resource including a tensor accelerator to perform optimized computation of sparse submatrix operations. One embodiment provides a parallel processor comprising a processing cluster coupled with the cache memory. The processing cluster includes a plurality of multiprocessors coupled with a data interconnect, where a multiprocessor of the plurality of multiprocessors includes a tensor core configured to load tensor data and metadata associated with the tensor data from the cache memory, wherein the metadata indicates a first numerical transform applied to the tensor data, perform an inverse transform of the first numerical transform, perform a tensor operation on the tensor data after the inverse transform is performed, and write output of the tensor operation to a memory coupled with the processing cluster.

Embodiments described herein include, software, firmware, and hardware logic that provides techniques to perform arithmetic on sparse data via a systolic processing unit. Embodiment described herein provided techniques to detect zero value elements within a vector or a set of packed data elements output by a processing resource and generate metadata to indicate a location of the zero value elements within the plurality of data elements.

A processing apparatus is described herein that includes a general-purpose parallel processing engine comprising a matrix accelerator including one or more systolic arrays, at least one of the one or more systolic arrays comprising multiple pipeline stages, each pipeline stage of the multiple pipeline stages including multiple processing elements, the multiple processing elements configured to perform processing operations on input matrix elements based on output sparsity metadata. The output sparsity metadata indicates to the multiple processing elements to bypass multiplication for a first row of elements of a second matrix and multiply a second row of elements of the second matrix with a column of matrix elements of a first matrix.

Embodiments are generally directed to cache structure and utilization. An embodiment of an apparatus includes one or more processors including a graphics processor; a memory for storage of data for processing by the one or more processors; and a cache to cache data from the memory; wherein the apparatus is to provide for dynamic overfetching of cache lines for the cache, including receiving a read request and accessing the cache for the requested data, and upon a miss in the cache, overfetching data from memory or a higher level cache in addition to fetching the requested data, wherein the overfetching of data is based at least in part on a current overfetch boundary, and provides for data is to be prefetched extending to the current overfetch boundary.

One embodiment provides a graphics processor comprising a base die including a plurality of chiplet sockets and a plurality of chiplets coupled with the plurality of chiplet sockets, at least one of the plurality of chiplets including a plurality of processing elements and a matrix accelerator coupled with the plurality of processing elements, the matrix accelerator having circuitry to perform a matrix multiply accumulate operation on matrix data having a tiled memory layout.

Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. The matrix multiply unit may include cells arranged in columns of the systolic array. Two chains of weight shift registers per column of the systolic array are in the matrix multiply unit. Each weight shift register is connected to only one chain and each cell is connected to only one weight shift register. A weight matrix register per cell is configured to store a weight input received from a weight shift register. A multiply unit is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result.

A processing apparatus described herein includes a general-purpose parallel processing engine comprising a systolic array having multiple pipelines, each of the multiple pipelines including multiple pipeline stages, wherein the multiple pipelines include a first pipeline, a second pipeline, and a common input shared between the first pipeline and the second pipeline.

Methods and apparatus for performing machine learning tasks, and in particular, to a neural-network-processing architecture and circuits for improved handling of partial accumulation results in weight-stationary operations, such as operations occurring in compute-in-memory (CIM) processing elements (PEs). One example PE circuit for machine learning generally includes an accumulator circuit, a flip-flop array having an input coupled to an output of the accumulator circuit, a write register, and a first multiplexer having a first input coupled to an output of the write register, having a second input coupled to an output of the flip-flop array, and having an output coupled to a first input of the first accumulator circuit.

On-circuit activity monitoring may be performed to modify integrated circuit processing. An activity monitor may be implemented on an integrated circuit to monitor activity measurements of processing data at another portion of the integrated circuit. A change to activity measurements may be detected and cause the activity monitor to modify the rate at which data enters the other portion of the integrated circuit for processing.

Embodiments described herein provide techniques to facilitate instruction-based control of memory attributes. One embodiment provides a graphics processor comprising a processing resource, a memory device, a cache coupled with the processing resources and the memory, and circuitry to process a memory access message received from the processing resource. The memory access message enables access to data of the memory device. To process the memory access message, the circuitry is configured to determine one or more cache attributes that indicate whether the data should be read from or stored the cache. The cache attributes may be provided by the memory access message or stored in state data associated with the data to be accessed by the access message.

The invention discloses an insulation defect feature detection method and system based on a switch cabinet, and relates to the field of insulation defect feature detection, and the method comprises the steps: installing a sensor array in the switch cabinet, collecting an original data stream of a multi-modal space-time physical field, and generating a reconstructed four-dimensional space-time field tensor through a multi-modal space-time cooperative sensing network; based on the four-dimensional space-time field tensor, generating an enhanced sample set conforming to a physical rule through physical regularization; generating an optimized detection parameter combination through gradient-evolution collaboration based on the enhanced sample set conforming to the physical law; digital twin enhanced closed-loop adaptive optimization is adopted, and the optimized detection parameter combination is converted into verified stable parameter configuration; according to the verified stable parameter configuration, real-time optimization parameters are obtained through embedded real-time acceleration optimization and solidification; the detection parameters are effectively adjusted and optimized by adopting a multi-target covariance evolutionary strategy to optimize a parameter combination process based on a space-time gradient characteristic spectrum.