Training convolutional neural networks on graphics processing units

Abstract

A convolutional neural network is implemented on a graphics processing unit. The network is then trained through a series of forward and backward passes, with convolutional kernels and bias matrices modified on each backward pass according to a gradient of an error function. The implementation takes advantage of parallel processing capabilities of pixel shader units on a GPU, and utilizes a set of start-to-finish formulas to program the computations on the pixel shaders. Input and output to the program is done through textures, and a multi-pass summation process is used when sums are needed across pixel shader unit registers.

Description

Background technique

[0001] Neural Networks

[0002] Certain computer problems, such as character recognition and image recognition, can be solved well with machine learning techniques, mainly using neural networks. A neural network is a class of algorithms based on the concept of interconnected "neurons". In a typical neural network, neurons have multiple data values, and each data value affects the value of a connected neuron according to a predefined connection strength, and it is judged whether the total connection for each specific neuron reaches the predetermined value. defined threshold. By determining the appropriate connection strength and threshold (a process also known as "training"), the neural network can effectively recognize graphics and characters. Often, these neurons are grouped into multiple "layers" to make the connections between groups more apparent for each computation of a data value.

[0003] Figure 1 illustrates a simplified neural network block d...

Examples

Embodiment Construction

[0019] The following description relates to training a convolutional neural network on a graphics processing unit ("GPU") architecture for handwriting recognition. The GPU repeatedly performs forward and backward passes on the input data, modifying and optimizing the matrices that comprise the neural network in each pass. Many of the methods described here have been designed to take advantage of the efficiency of GPUs and use pixel shader programs designed to execute efficiently on GPUs.

[0020] 1. GPU architecture

[0021] The methods described here are implemented in a graphics processing unit. A graphics processing unit as shown in FIG. 1 illustrates a brief description of a conventional GPU architecture 300 . In one implementation, the GPU architecture corresponds to GPU 815 shown in FIG. 8 . Display data 305 , which describes the geometry of the data to be rendered, is input into vertex shader unit 310 to generate a polygonal representation of its geometric form. The...

Background technique

[0001] Neural Networks

[0002] Certain computer problems, such as character recognition and image recognition, can be solved well with machine learning techniques, mainly using neural networks. A neural network is a class of algorithms based on the concept of interconnected "neurons". In a typical neural network, neurons have multiple data values, and each data value affects the value of a connected neuron according to a predefined connection strength, and it is judged whether the total connection for each specific neuron reaches the predetermined value. defined threshold. By determining the appropriate connection strength and threshold (a process also known as "training"), the neural network can effectively recognize graphics and characters. Often, these neurons are grouped into multiple "layers" to make the connections between groups more apparent for each computation of a data value.

[0003] Figure 1 illustrates a simplified neural network block d...

Examples

Embodiment Construction

[0019] The following description relates to training a convolutional neural network on a graphics processing unit ("GPU") architecture for handwriting recognition. The GPU repeatedly performs forward and backward passes on the input data, modifying and optimizing the matrices that comprise the neural network in each pass. Many of the methods described here have been designed to take advantage of the efficiency of GPUs and use pixel shader programs designed to execute efficiently on GPUs.

[0020] 1. GPU architecture

[0021] The methods described here are implemented in a graphics processing unit. A graphics processing unit as shown in FIG. 1 illustrates a brief description of a conventional GPU architecture 300 . In one implementation, the GPU architecture corresponds to GPU 815 shown in FIG. 8 . Display data 305 , which describes the geometry of the data to be rendered, is input into vertex shader unit 310 to generate a polygonal representation of its geometric form. The...

Images