Using neural network accelerators to process continuous input

Abstract

A hardware accelerator is capable of storing input data for each of a plurality of processing time steps in a plurality of memory storage areas in one or more memories on the accelerator, the plurality of processing time steps being used to process a serialized input to a machine learning model. For each processing time step, the following operations are performed. The accelerator is capable of accessing a current value of a counter stored in a register within the accelerator to identify the processing time step. The accelerator is capable of determining one or more memory storage areas based on the current value of the counter, the memory storage areas storing the input data for the processing time step. The accelerator is capable of facilitating access of the input data for the processing time step from the one or more memory storage areas to at least one processor coupled to the one or more memory storage areas. The accelerator is capable of incrementing the current value of the counter stored in the register.

Description

Technical Field

[0001] The subject matter described herein relates generally to hardware accelerators, and more particularly to hardware accelerators configured to quickly look up data stored within a memory of the hardware accelerator. Background Art

[0002] A hardware accelerator is computer hardware specifically designed to perform some operations more efficiently than would be possible in software running on a general-purpose central processing unit (CPU). Such hardware accelerators are typically associated with a host device (e.g., a CPU) that performs some operations, while the hardware accelerator performs other operations. When the hardware accelerator needs to perform an operation, it frequently communicates with the host device to, for example, receive instructions and input data or the identification of a specific memory storage area where the input data is stored. This frequent communication can disadvantageously cause computational delays. Summary of the Invention

[0003] In one aspect, a hardware accelerator can store input data for each of a plurality of processing time steps in a plurality of memory storage areas in one or more memories on the hardware accelerator, the plurality of processing time steps being used to process a sequenced input to a machine learning model (e.g., a recurrent neural network). For each of the plurality of processing time steps, the following operations can be performed. The hardware accelerator can access a current value of a counter stored in a register within the hardware accelerator. The current value of the counter can identify the processing time step. The hardware accelerator can determine one or more memory storage areas based on the current value of the processing time step counter, the one or more memory storage areas storing the input data for the processing time step. The hardware accelerator can facilitate access of the input data for the processing time step from the one or more memory storage areas to at least one processor coupled to the one or more memory storage areas. The hardware accelerator can increment the current value of the counter stored in the register.

[0004] In some variants, one or more of the following can also be implemented individually or in any feasible combination. Determining the one or more memory storage areas storing input data for the processing time step can include: retrieving, by the hardware accelerator, a value of a stride associated with a machine learning model (e.g., a recurrent neural network); calculating, by the hardware accelerator and based on the current value of the counter and the value of the stride, the values ​​of at least two edges of the input data for the processing time step; and determining, by the hardware accelerator and based on the values ​​of the at least two edges, the one or more memory storage areas storing the input data for the processing time step. The hardware accelerator can generate a mapping of each memory storage area and the ends of the one or more memory storage areas. The hardware accelerator can store the mapping in a register within the hardware accelerator. The ends of the one or more memory storage areas encompass the at least two edges. Calculating the values ​​of the edges can include multiplying, by the hardware accelerator, the current value of the counter by the value of the stride.

[0005] The hardware accelerator can receive a single instruction from the central processing unit for each of the plurality of processing time steps. In response to receiving the single instruction, the hardware accelerator can perform at least determining the one or more storage areas and facilitating access to the at least one processor of input data for the processing time step. The hardware accelerator can store the single instruction in another memory within the hardware accelerator. The hardware accelerator and the central processing unit can be embedded in a mobile phone.

[0006] The hardware accelerator can receive input data for each of the plurality of processing time steps from the central processing unit before storing the input data. Storing the input data for each of the plurality of processing time steps can include transferring the input data for each of the plurality of processing time steps to the plurality of memory storage areas. A controller can transfer the input data for each of the plurality of processing time steps to the plurality of memory storage areas via at least one bus.

[0007] At least one processor and one or more memory storage areas are present within a single computing unit in the plurality of computing units. Input data can be separate and / or different for each processing time step in the plurality of processing time steps. The hardware accelerator can store output generated by a machine learning model (e.g., a recurrent neural network) for each of the plurality of processing time steps in another memory within the hardware accelerator. The hardware accelerator can collectively transmit the output for each of the plurality of processing time steps after the plurality of processing time steps.

[0008] In another aspect, a non-transitory computer program product is described that can store instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform the operations described herein. In yet another aspect, a system can include at least one programmable processor and a machine-readable medium storing instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform the operations described herein. In some aspects, an article of manufacture can include computer-executable instructions stored on a non-transitory computer-readable medium that, when executed by a computer, cause the computer to perform the operations described herein.

[0009] The subject matter described herein provides numerous advantages. For example, the architecture described herein allows the accelerator to minimize communication with a host device (e.g., a CPU). This minimization of communication can advantageously minimize computational latency. Furthermore, the accelerator describes computations, such as tensor computations, as being distributed across multiple compute units. Configuring each compute unit to perform a specific tensor computation can advantageously allow for accelerated tensor traversal and computation of tensor operations, for example by reducing the number of instructions. This acceleration advantageously improves computational efficiency. Furthermore, the input to a machine learning model (e.g., a neural network) includes a sequence of multiple inputs—each corresponding to a different processing time step—and the accelerator receives all inputs for all processing time steps immediately from the host device via a single instruction that instructs the accelerator to perform a specific operation on all input data for all processing steps (including all the inputs mentioned above). Having a single instruction from the host device for all processing steps is superior to having separate instructions for each processing step (corresponding to potentially different slices of all input data). For example, the computational bandwidth of the accelerator can be increased by reducing the number of instructions that the accelerator needs to execute when traversing a tensor to perform one or more computations.

[0010] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below.Other features and advantages of the subject matter described herein will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An accelerator is shown that is configured to perform specific operations of a neural network and a host device (e.g., a central processing unit (CPU)).

[0012] Figure 2 The illustrated memory, which can be one of the memories on the accelerator, includes a first memory storage area and a second memory storage area.

[0013] Figure 2A Figure 1 shows the computing unit of an accelerator.

[0014] Figure 3 The diagram shows a table of some types of operations that can be specified (ie, encoded) by an instruction received from a host device via an accelerator.

[0015] Figure 4 A diagram illustrates a mapping of specific memory storage regions and corresponding memory on the accelerator and the ends of input data.

[0016] Figure 5 Illustrated is a method performed by an accelerator to access a counter to determine one or more memory storage areas storing input data for a current processing step of a RNN.

[0017] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0018] Figure 1 The diagram illustrates an accelerator 102 configured to execute specific operations of a neural network and a host device (e.g., a central processing unit (CPU)) 104. The neural network can be a recurrent neural network (RNN). Although the neural network is described as an RNN in this document, in some embodiments, the neural network can be any neural network capable of processing a series of inputs over multiple time steps, such as a neural network based on transformer self-attention, a convolutional neural network, etc. The RNN can be a long short-term memory (LSTM) network, a gated rectifier unit (GRU), or any other recurrent neural network that maintains and updates internal state while processing a series of inputs over multiple time steps. The accelerator 102 can also be referred to as a hardware accelerator. The accelerator 102 and the host device 104 can be embedded in a computing device, such as a mobile phone or tablet. The accelerator 102 can have a controller 106 and multiple computing units, such as 108, 110, 112, 114, 116, 118, 120, and 122. Although eight computational units 108-122 are shown, in alternative embodiments, the accelerator 102 can have any other number of computational units, such as any number between two and six. Each computational unit 108-122 can have at least one programmable processor 124-138 and at least one memory 140-154. The accelerator 102 can also have other memories, such as an instruction memory 156, a data memory 158, a memory 160 configured to store counters, and a memory 162 configured to store outputs. Each of the memories 140-154 can have several memory storage areas. Each memory storage area can be configured to store therein (e.g., weights for an RNN) as described below by Figure 2 The following is the input data of the interpretation. Figure 2AOne example of a computing unit (ie, one of computing units 108 - 122 ) is explained in more detail.

[0019] To perform specific operations of the RNN, the controller 106 receives input data for the RNN and instructions specifying the operations to be performed by the accelerator 102 from the host device 104. The input data can include the weights of the RNN and data on which the RNN is executed to generate inferences. The input to the RNN can include a series of multiple inputs, and each input can correspond to a different processing time step. In some unfavorable embodiments, the controller 106 can receive slices of all input data for each processing time step of the RNN (i.e., the input corresponding to the processing time step) from the host device 104 one by one. However, here, the controller 106 receives all input data for the RNN (i.e., all inputs for all processing steps) from the host device 104 all at once - this minimizes communication between the accelerator 102 and the host device 104, thereby advantageously minimizing latency and maximizing computational efficiency.

[0020] The instruction is a single instruction that instructs the accelerator 102 to perform a specific RNN operation on the input data for all processing time steps. A single instruction with all input data for all processing time steps from the host device 104 is superior to multiple instructions corresponding to individual processing time steps, thereby minimizing communication between the accelerator 102 and the host device 104, which in turn advantageously reduces / minimizes computational latency and, therefore, maximizes computational efficiency. The specific RNN operations described above can include tensor operations, data memory access operations, guard operations, and / or other operations. Figure 3 Describes an example of various operations.

[0021] The controller 106 can transmit the received input data for all processing time steps in a serial manner to the memories 140-154 for storage of the data via at least one bus. The controller 106 can then generate each of the memories 140-154 and a memory storage area within each memory storing input data (e.g., Figure 2 ) and the end value of the memory storage area, as shown below by Figure 4 The controller 106 can store this mapping in the data store 158 .

[0022] The controller 106 can store a counter indicating a current processing time step of a plurality of processing time steps for processing by the RNN in the memory 160. The controller 106 can increment the counter after each processing time step of the RNN.

[0023] For each processing time step of the RNN, the controller 106 can perform the following operations. The controller 106 can access the current value of the counter from the memory 160 to identify the processing time step. The controller 106 can determine one or more memory storage areas within the one or more memories 140-154 that store input data for the processing time step based on the current value of the counter, as explained in further detail below. The controller 106 can facilitate access of the input data for the processing time step from the one or more memory storage areas to at least one processor coupled to the one or more memory storage areas. The controller 106 can then increment the current value of the counter stored in the memory 130, also as described above.

[0024] The storage of the counters and mapping (described above) on the accelerator 102 is advantageous because it prevents the need for the accelerator to communicate with the host device to obtain multiple instructions within a processing time step, thereby minimizing computational latency.

[0025] The controller 106 can determine one or more memory storage areas within the memories 140-154 that store input data for processing a time step according to the following. The controller 106 can retrieve the stride value associated with the recurrent neural network from a compilation result generated by a compiler (which can also be embedded within a computing device in which the accelerator 102 and / or the host device 104 resides) and provided by the compiler to the accelerator 102. The compilation result can be machine code in a low-level computer language that can be interpreted by the accelerator 102. The controller 106 can determine the values ​​of at least two edges of the input data for processing the time step based on the current value of the counter and the value of the stride. More specifically, the controller 106 can calculate each edge by multiplying the current value of the counter by the value of the stride. The controller 106 can use a mapping stored in the data memory 158 and determine one or more memory storage areas based on the values ​​of the at least two edges that store the input data for processing the time step. For example, if the left edge of the input data for a processing time step has a value of 103 and the right edge of the input data has a value of 147, the controller can use Figure 4 The mapping determines that the input data for the processing time step is stored in the first memory storage area 204 of the memory 144 (which is in the third computing unit 112).

[0026] The controller 106 can store the output generated by the recurrent neural network for each of the plurality of processing time steps in the memory 162. The controller 106 can then transmit the outputs for all processing time steps together, rather than transmitting the output generated by the RNN for each processing time step individually. This transmission of outputs for all processing time steps (rather than multiple outputs corresponding to each processing time step) minimizes communication between the accelerator 102 and the host device 104, thereby advantageously minimizing computational latency and, therefore, maximizing computational efficiency. In an alternative embodiment, the controller 106 can transmit the output for a processing time step after the plurality of processing time steps have been completed (e.g., the controller 106 can transmit the outputs for three processing time steps together to the host device 104).

[0027] Figure 2 Memory 202 is shown, which can be any of memories 140-154, including a first memory storage area 204 and a second memory storage area 206. The first memory storage area 204 can be configured to store input data for the RNN. The second memory storage area 206 can be configured to store weights for the RNN. Memory storage areas can also be referred to as registers or banks.

[0028] The memory storage area 204 can include one or more narrow memories, and the memory storage area 206 can include one or more wide memories. The narrow and wide names are attributes of the corresponding memories and generally refer to the width of the memory in bits and / or bytes (and in alternative embodiments, to approximate widths).

[0029] Figure 2A The diagram shows an example of a computation unit 200 that performs one or more RNN computations. The computation unit 200 can correspond to any of the computation units 108-122. Each computation unit 200 is a self-contained computation unit that is configured to execute instructions independently of other corresponding computation units within the computation units 108-122. Each computation unit 200 can execute two types of instructions, tensor operation (also known as TensorOp) instructions and direct memory access operation (also known as DMAOp) instructions. Each instruction type includes computation operations associated with the deep loop nesting of the RNN, and therefore each instruction type is typically executed in multiple stages to ensure completion of all loop iterations.

[0030] Different instruction types are executed by independent control units within the compute block 200, which synchronize on data via synchronization tag controls managed within the compute block 200. The synchronization tag controls manage the parallelism between the execution of different instruction types within the compute block 200. Each computational operation associated with each instruction type is executed in a strict issue order (i.e., first-in, first-out). With respect to two instruction types, TensorOP and DMAOp, there is no ordering preservation between these different instruction types, and each type is handled by the compute unit 200 as a separate control thread.

[0031] Regarding data flow construction, the computing unit 200 generally includes a data path 201 and a data path 205, each of which provides a communication path for data flow into and out of the computing unit 200. The computing units 108-154 can be connected via three different data bus structures, which can be arranged in a ring configuration - a CSR bus, an instruction bus, and a ring bus. Data path 205 corresponds to the instruction bus, while data path 201 generally corresponds to one of the CSR bus and the ring bus. Data path 201 includes a ring output 203 that provides an output path for data leaving the computing unit 200 and a ring input 210 that provides an input path for data entering the computing unit 200.

[0032] The compute unit 200 also includes a TensorOp control 212 including a TensorOp tensor traversal unit (TTU) 226, and a DMAOp control 208 including a DMAOp TTU 228. The TensorOp control 212 generally manages writes to and reads from TensorOp TTU registers 232 and performs traversal operations for execution by the TensorOp TTU 226. Similarly, the DMAOp control 208 generally manages writes to and reads from DMAOp TTU registers 234 and performs traversal operations for execution by the DMAOp TTU 228. The TTU registers 232 include an instruction buffer for storing one or more instructions including operations to be performed by the TensorOp TTU 226 when the instructions are executed by the TensorOp control 212. Likewise, the TTU registers 234 include an instruction buffer for storing one or more instructions comprising operations to be performed by the TTU 208 when the instructions are executed by the DMAOp control 208. As described further below, the TTU is used by the compute unit 200 to traverse array elements of one or more tensors, which typically reside in the narrow memory 204 and the wide memory 206.

[0033] In some embodiments, certain instructions for execution by the compute block 200 arrive at the block via the data path 205 (i.e., part of the instruction bus). The compute unit 200 examines the header bitmap to determine the instruction type (TensorOp or DMAOp) and the instruction subtype (read operation or write operation). The instruction(s) received by the compute unit 200 are then written to a specific instruction buffer depending on the instruction type. Generally, the instructions are received and stored (i.e., written to the buffer) before being executed by the components of the compute unit 200. The instruction buffers (i.e., TensorOp TTU registers 232 and DMAOp TTU registers 234) can each include a first-in, first-out (FIFO) control scheme that prioritizes the consumption (execution) of one or more related instructions.

[0034] A tensor is a multidimensional geometric object, and example multidimensional geometric objects include matrices and data arrays. An algorithm including at least some computational operations of an RNN can be executed by the computation unit 200 to perform tensor computations by iterating one or more nested loops to traverse an N-dimensional tensor. In an example computational process, each nest of loop nests can be responsible for traversing a specific dimension of the N-dimensional tensor. As described herein, the TensorOp control 212 typically performs one or more tensor operations that drive the order in which dimensional elements of a particular tensor structure are traversed and accessed to complete the computation defined by the deeply nested loops.

[0035] The computing unit 200 also includes a narrow memory 204 and a wide memory 206, also as shown by Figure 2 As discussed. Narrow nomenclature and wide nomenclature generally refer to the size of the width (bits / bytes) of the memory cells of the narrow memory 204 and the wide memory 206. In some embodiments, the narrow memory 204 includes memory cells that each have a size or width of less than 16 bits, and the wide memory 206 includes memory cells that each have a size or width of less than 32 bits. Generally, the compute unit 200 receives input activations via the data path 205, and the DMA control 208 performs operations that write the input activations to the narrow memory 204. Similarly, the compute unit 200 receives parameters (weights) via the data path 201, and the DMA control 208 performs operations that write the parameters to the wide memory 206. In some embodiments, the narrow memory 204 can include a memory arbiter, which is typically used in a shared memory system to decide for each memory cycle which control device (e.g., the TensorOp control 212 or the DMAOp control 208) will be allowed to access the shared memory cell of the narrow memory 204.

[0036] Compute unit 200 also includes an input activation bus 216 and a MAC array 214 comprising a plurality of cells, each of which includes a MAC operator 215 and a sum register 220. Generally, MAC array 214 uses MAC operators 215 and sum registers 220 to perform tensor computations across multiple elements, including arithmetic operations related to dot product computations. Input activation bus 216 provides a data path in which input activations provided by narrow memory 204 are accessed individually by each MAC operator 215 in MAC array 214. Thus, based on the propagation of input activations, each individual MAC operator 215 of a particular element will receive an input activation. The arithmetic operations performed by the MAC operators of MAC array 214 typically involve multiplying the input activations provided by narrow memory 204 with parameters accessed from wide memory 206 to produce a single output activation value.

[0037] During an arithmetic operation, the partial sums may be accumulated and stored in corresponding, for example, sum registers 220, or written to wide memory 206 and re-accessed by a particular element of MAC array 214 to complete a subsequent multiplication operation. Tensor computations can be described as having a first part and a second part. The first part is performed when the multiplication operation produces an output activation, for example, by multiplying the input activation by a parameter to generate the output activation. The second part includes applying a nonlinear function to the output activation and is completed when the output activation is written to narrow memory 204 after the function is applied.

[0038] The compute block 200 also includes an output activation bus 218, a nonlinear unit (NLU) 222 including an output activation pipeline 224, an NLU control 238, and a reference map 230 indicating core properties of the components in the compute block 200. The reference map 230 is not included in the compute block 200 for clarity of illustration. Core properties include whether a particular component is a unit, a storage device, an operator, a control device, or a data path. Generally, when completing the first part of a tensor computation, output activations are provided from the MAC array 214 to the NLU 222 via the output activation bus 218. After arriving at the NLU 222, data specifying an activation function received via the activation pipeline 224 is applied to the output activations, and the output activations are then written to the narrow memory 204. In some embodiments, the output activation bus 218 includes at least one pipelined shift register 236, and completing the second part of the tensor computation includes using the shift register 236 of the activation bus 218 to shift the output activations toward the narrow memory 204.

[0039] For example, regarding the computation of the dot product of two multidimensional data arrays, the MAC array 214 provides robust single instruction, multiple data (SIMD) functionality for a single compute unit 200. SIMD generally means that all parallel units (multiple MAC operators 215) share the same instruction (based on deep loop nesting), but each MAC operator 215 executes the instruction on a different data element. In a basic example, element-wise addition of arrays [1, 2, 3, 4] and [5, 6, 7, 8] to obtain array [6, 8, 10, 12] in a single loop would typically require four arithmetic units to perform the operation on each element. By using SIMD, the four units can share the same instruction (e.g., "add") and perform the computation in parallel. Thus, the accelerator 102 and compute unit 200 provide enhanced speedup and parallelism in tensor computations over existing approaches.

[0040] In one example, a single instruction can be provided by the controller 106 to multiple compute units 200 (e.g., compute units 108-122) for consumption by multiple MAC arrays 214. In general, a neural network layer can include multiple output neurons, and the output neurons can be partitioned so that tensor computations associated with a subset of the output neurons can be assigned to specific blocks of the block sets 112, 114. Each block in the block sets 112, 114 can then perform related tensor computations on a different set of neurons of a given layer. The compute unit 200 can thus provide at least two forms of parallelism: (1) one form includes partitioning the output activations among multiple compute units 108-122 (corresponding to subsets of output neurons); and (2) another form includes simultaneous computations (with a single instruction) based on multiple subsets of the partitioned output neurons among the compute units 108-122. A compiler coupled to the accelerator 102 (which may also be embedded within the accelerator 102 and / or a computing device residing on the host device 104) can determine and indicate to the accelerator 102 a specific target block for each of the neural network inputs. For example, the compiler generates runtime code to determine one or more computational units (i.e., one or more of the computational units 108-122) having a relevant portion of the input for each particular processing time step.

[0041] Figure 3The diagram shows a table 302 of some types of operations that can be specified (i.e., encoded) by an instruction received by the accelerator 102 from the host device 104. The table 302 can be stored in a memory accessible by the accelerator 102 and the host device 104. For all processing time steps corresponding to the input received continuously by the RNN, the accelerator 102 can receive a single instruction from the host device 104, rather than multiple instructions, as described above. The instructions received by the accelerator 102 can include operation codes (also called opcodes) 304 for various operations. The accelerator 102 can use the table 302 to infer the operation specified by the opcode 304. The table 302 indicates the operation type 306 and description 308 specific to the operation of each opcode 304.

[0042] Instructions can have a standardized structure that can include one or more opcodes 304 to specify one or more operations to be performed by a particular computation unit to achieve a particular computation order.The opcode for each operation can be different from other opcodes.

[0043] Operation types 306 can include tensor operations, direct memory access operations, and / or guard operations, as shown, and / or other possible operations not shown. Controller 106 can transmit the relevant portions of the instruction to the electronic components, such as the computing units that need to execute those portions of the instruction. For example, an operation that is designated for a particular computing unit to perform the operation needs to be performed by the designated computing unit.

[0044] If data is moved between two memory resources of unequal width (e.g., wide memory and narrow memory), the transfer size of the data must be a multiple of the wider resource. For example, if a direct memory access operation moves data to / from wide memory, the size of the transferred data must be a multiple of the width of the wide memory. Similarly, for direct memory access operations that move data to / from narrow memory, the size of the data transfer must be a multiple of the width of the narrow memory. The accelerator 102 can identify the operation that must be performed from the tensor operation and can identify how the data is copied from the DMA operation, including the source and destination memory addresses, the size of the data transfer, and the method of synchronizing the copying of the data with the computation technique. Computation can begin as soon as the expected portion of the data is in memory.

[0045] Figure 41. A map 402 is shown of memory storage areas (e.g., first memory storage area 204 or second memory storage area 206) and corresponding memories (i.e., one of memories 140-154) 404 and ends 406 of those memory storage areas. In the example shown, it is assumed that each of memories 140-154 has the same storage capacity and therefore stores the same amount of data. In alternative embodiments, at least some of memories 140-154 can have different storage capacities, which in turn would allow Figure 4 To maximize storage efficiency, the input data are advantageously stored continuously (rather than, for example, storing input data corresponding to each processing time step in a separate memory).

[0046] When the RNN is to process data, the controller 106 can calculate the edge of the input data for each processing time step based on the current value of the counter and the value of the stride, as described above. For example, the controller 106 can calculate each edge by multiplying the current value of the counter and the value of the stride. The controller 106 can then use the value of the edge to determine one or more memories 204 using the mapping 402, which store the input data for the processing time step. For example, if the value of the left edge of the input data for the processing time step is 103 and the value of the right edge of the input data is 147, the controller can use Figure 4 The mapping determines that the input data for the processing time step is stored in the first memory storage area 204 within the memory 144.

[0047] Once the controller identifies which specific memory portion is required for the input data, it extracts the input data from that specific memory storage area. The controller identifies a specific memory address from instructions generated and provided by the compiler, and retrieves the input data for that specific memory storage area from that memory address. The compiler generates such instructions during compile time to identify the source location, which includes a block and the memory address of that block with the appropriate input data. Once the source memory is identified, the generated instructions write information to registers on the memory. Since registers are part of the instructions, the accelerator 102 executes the instructions generated by the compiler to obtain the source memory address and data information, such as the size of the data to be transferred.

[0048] Figure 5 The diagram illustrates a method executed by the accelerator 102 to access a counter to determine one or more memory storage areas storing input data for a current processing time step of the RNN. The controller 106 can store input data for each of a plurality of processing time steps processed by the RNN in one or more memory storage areas within the memories 140-154.

[0049] The controller 106 can then perform the following for each processing time step. At 502, the controller 106 can access the current value of a counter stored in the memory 160. The current value of the counter can identify the processing time step. At 504, the controller 106 can determine one or more specific memory storage areas based on the current value of the counter, which memory storage areas store input data for the processing time step. At 506, the controller 106 can facilitate access of the input data for the processing time step from the one or more specific memory storage areas to at least one of the processors 124-138 coupled to the one or more memory storage areas. For example, the controller 106 can send an instruction to a source block (i.e., a block having data) that can cause the source block to send data to a target block (i.e., a block that needs to perform initial calculations for the time step). At 508, the controller can increment the current value of the counter stored in the memory 160.

[0050] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuit systems, in tangibly implemented computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more thereof. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions implemented on a tangible non-transitory program carrier for execution by a data processing device or to control the operation of a data processing device. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagation signal, for example, a machine-generated electrical signal, optical information, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof.

[0051] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of a programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, for example, files that store portions of one or more modules, subroutines, or code. A computer program can be deployed to execute on one or more computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0052] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and the apparatus can also be implemented as, a special purpose logic circuitry, such as an FPGA (field programmable gate array), an ASIC (application-specific integrated circuit), a GPGPU (general purpose graphics processing unit), or some other type of processor.

[0053] The computer that is suitable for executing computer programs includes, by way of example, a central processing unit that can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other type of microprocessor. Generally, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operably coupled to receive data from or transmit data to one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not necessarily have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0054] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be implemented by, or incorporated in, special purpose logic circuitry.

[0055] The embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes an intermediate component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser, via which a user can interact with the embodiments of the subject matter described in this specification, or any combination of one or more such back-end components, intermediate components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0056] A computing system can include clients and servers. A client and server are generally remote from each other and typically interact via a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0057] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the content claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in a combination of a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination can be excluded from the combination in some cases, and the claimed combination can be guided to a sub-combination or a variation of the sub-combination.

[0058] Similarly, when operations are depicted in a particular order in a figure, this should not be understood as requiring that such operations be performed in the particular order shown or in a serialized order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0059] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A method for processing serialized input using a neural network accelerator, comprising: storing, by a hardware accelerator, input data for each of a plurality of processing time steps in a plurality of memory storage areas in one or more memories on the hardware accelerator, the plurality of processing time steps being used to process a serialized input to a machine learning model; For each processing time step in the plurality of processing time steps: accessing, by the hardware accelerator, a current value of a counter stored in a register within the hardware accelerator, the current value of the counter identifying the processing time step; determining, by the hardware accelerator, one or more memory storage areas to store the input data for the processing time step based on the current value of the processing time step counter; facilitating, by the hardware accelerator, access to the input data for the processing time step in the one or more memory storage areas from at least one processor coupled to the one or more memory storage areas; as well as The current value of the counter stored in the register is incremented by the hardware accelerator.

2. The method according to claim 1, wherein Determining the one or more memory storage areas storing the input data for the processing time step comprises: retrieving, by the hardware accelerator, a stride value associated with the machine learning model; calculating, by the hardware accelerator and based on the current value of the counter and the value of the stride, values ​​of at least two edges of the input data for the processing time step; and The one or more memory storage areas storing the input data for the processing time step are determined by the hardware accelerator based on the values ​​of the at least two edges.

3. The method according to claim 2, further comprising: generating, by the hardware accelerator, a mapping of each memory storage region and ends of the one or more memory storage regions; as well as storing, by the hardware accelerator, the mapping in a register within the hardware accelerator; Wherein, the end portions of the one or more memory storage areas cover the at least two edges.

4. The method according to claim 3, wherein: The calculation of the value of the edge comprises: The current value of the counter is multiplied by the hardware accelerator by the value of the stride.

5. The method according to claim 1, further comprising: receiving, by the hardware accelerator from a central processing unit, a single instruction for each of the plurality of processing time steps, wherein the hardware accelerator performs at least the determining of the one or more storage regions and the facilitating of the access of the input data to the at least one processor for the processing time step in response to the receipt of the single instruction.

6. The method according to claim 5, further comprising: The single instruction is stored by the hardware accelerator in another memory within the hardware accelerator.

7. The method according to claim 6, wherein: The hardware accelerator and the central processing unit are embedded in a mobile phone.

8. The method according to claim 1, further comprising: The input data for each of the plurality of processing time steps is received by the hardware accelerator from a central processing unit prior to the storing.

9. The method according to claim 1, wherein: The storing of the input data for each processing time step of the plurality of processing time steps comprises: The input data for each of the plurality of processing time steps is transferred by a controller of the hardware accelerator to the plurality of memory storage areas.

10. The method according to claim 9, wherein: The controller transmits the input data for each of the plurality of processing time steps to the plurality of memory storage areas via at least one bus.

11. The method according to claim 1, wherein The at least one processor and the one or more memory storage areas reside within a single computing unit of the plurality of computing units.

12. The method according to claim 1, wherein The input data is separate and different for each processing time step in the plurality of processing time steps.

13. The method according to claim 1, further comprising: storing, by the hardware accelerator, an output generated by the machine learning model for each of the plurality of processing time steps in another memory within the hardware accelerator; as well as The output for each of the plurality of processing time steps is transmitted collectively by the hardware accelerator after the plurality of processing time steps.

14. A non-transitory computer program product storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising: storing input data for each of a plurality of processing time steps in a plurality of memory storage areas in one or more memories, the plurality of processing time steps being used to process a serialized input to a machine learning model; For each processing time step in the plurality of processing time steps: accessing a current value of a counter stored in a register, the current value of the counter identifying the processing time step; determining one or more memory storage areas to store the input data for the processing time step based on the current value of the processing time step counter; facilitating access to the input data for the processing time step in the one or more memory storage areas from at least one processor coupled to the one or more memory storage areas; as well as The current value of the counter stored in the register is incremented.

15. The non-transitory computer program product of claim 14, wherein: Determining the one or more memory storage areas storing the input data for the processing time step comprises: retrieving a value of a stride associated with the machine learning model; calculating values ​​of at least two edges of the input data for the processing time step based on the current value of the counter and the value of the stride; and The one or more memory storage areas storing the input data for the processing time step are determined based on the values ​​of the at least two edges.

16. The non-transitory computer program product of claim 15, wherein: The operations further include: generating a mapping of each memory storage region and ends of the one or more memory storage regions; and storing the mapping in a register, Wherein, the end portions of the one or more memory storage areas cover the at least two edges.

17. The non-transitory computer program product of claim 16, wherein: The calculation of the value of the edge comprises: The current value of the counter is multiplied by the value of the stride.

18. The non-transitory computer program product of claim 14, wherein: The operations further include: receiving, from a central processing unit, a single instruction for each of said plurality of processing time steps, wherein said non-transitory computer program performs, in response to said receipt of said single instruction, at least said determining of said one or more storage areas and said facilitating of said access of said input data for said processing time step to said at least processor; and The single instruction is stored in another memory.

19. The non-transitory computer program product of claim 14, wherein: The operations further include: receiving said input data for each of said plurality of processing time steps from a central processing unit prior to said storing; wherein said storing of said input data for each of said plurality of processing time steps transfers said input data for each of said plurality of processing time steps to said plurality of memory storage areas.

20. The non-transitory computer program product of claim 14, wherein: The operations further include: storing in another memory an output generated by the machine learning model for each of the plurality of processing time steps; and The output for each of the plurality of processing time steps is transmitted collectively after the plurality of processing time steps.

Images