Parallel processing device and method for parallel multivalued reduction

Abstract

Methods, systems and apparatus, including computer programs encoded on computer storage media, for computing multi-valued reductions using parallel processing devices. One of the methods includes performing a parallel M-value reduction by a parallel processing unit of a parallel processing device. A plurality of initial reductions are performed serially, and each initial reduction operates on data in different corresponding register spaces of the at least M register spaces. Data is moved from the M register spaces such that all results from multiple initial reductions are in the same first register space. One or more subsequent reductions are performed in parallel to compute the M final values, each subsequent reduction operating only on data in said first register space.

Description

[0001] Cross References to Related Applications [0002] This application claims priority to U.S. Patent Application No. 15 / 839,637, filed December 12, 2017, the entire contents of which are hereby incorporated by reference. Background technique [0003] This specification relates to techniques for efficiently computing multivalued reductions in parallel using parallel processing hardware. [0004] A reduction is an operation that combines multiple values ​​into a single value. For example, reduction of 8 values ​​can be done by computing a single sum of 8 values. A reduction operation is typically performed by a parallel processing device, such as a graphics processing unit (GPU), to combine data computed by multiple threads executed by multiple independent processing units of the parallel processing device. [0005] The examples described in this specification will generally involve the independent processing unit being a streaming multiprocessor (SM) with multiple process...

AI Technical Summary

Problems solved by technology

[0010] However, performing multi-valued reductions usually requires serialization of multiple single-valued reductions

In many practical applications with extremely high throughput requirements, this limit is the processing bottleneck

For example, audio generative neural networks that model raw audio waveforms present significant computational challenges due to the fundamentally high-throughput nature of raw audio generation

Images