Inference device and method for performing inference upon multiple input data via neural network involving at least hidden state inheritance

Abstract

An inference device includes a storage device, processing circuitry, and memory. The storage device stores code to instruct the processing circuitry to execute a neural network model involving at least hidden state inheritance. The processing circuitry performs multiple inference operations on multiple input data points of the neural network model in parallel at multiple time steps. Each inference operation includes: performing a feature extraction operation on a single input data point to produce an extraction result; reading the inherited hidden state corresponding to a previous time step from memory; performing a matrix operation based on the extraction result and the inherited hidden state to produce a current hidden state corresponding to the current time step, wherein the time period required for the matrix operation is a positive integer; and performing an inference operation on one of multiple branches based on the current hidden state to produce an inference result.

Claims

1. An inference apparatus comprising: a storage device for storing code; a processing circuit for loading and executing the code, wherein the code instructs the processing circuit to execute a neural network model involving at least hidden state inheritance, and the processing circuit is further configured to perform multiple inference operations on multiple input data of the neural network model in parallel at multiple time steps; a memory for storing multiple hidden states corresponding to the multiple time steps; wherein each of the multiple inference operations comprises: performing a feature extraction operation on one of the multiple input data to produce an extraction result; and reading an inherited hidden state corresponding to a previous time step from the memory. A matrix operation is performed based on the extraction result and the inherited hidden state to produce a current hidden state corresponding to a current time step, wherein the current hidden state is written into the memory for the inheritance of the hidden state, a difference between the previous time step and the current time step is a time offset value, the time offset value is a time cycle count required to perform the matrix operation, and the time offset value is a positive integer; and an inference operation is performed based on the current hidden state for one of the multiple branches to produce an inference result.

2. The inference apparatus as described in claim 1, wherein the processing circuit performs the multiple inference operations on the multiple input data of the neural network model in parallel at multiple time steps by means of multiple pipeline stages.

3. The inference apparatus as described in claim 1, wherein the neural network model involves the inheritance and sharing of hidden states to support inference operations for any one of the plurality of branches.

4. The inference apparatus as described in claim 1, wherein the plurality of branches are configured to share the current hidden state to avoid any duplicate calculations.

5. The inference device as described in claim 1, wherein the current hidden state is broadcast to the plurality of branches to allow the plurality of branches to share the current hidden state.

6. The inference apparatus as described in claim 1, wherein the neural network model is configured to operate in one of a plurality of predetermined modes, and the plurality of predetermined modes includes at least one power-saving mode for disabling at least one of the plurality of branches to reduce overall power consumption.

7. The inference device as described in claim 1, wherein the neural network model is configured to selectively clock-gated at least one of the plurality of branches in response to a change in context to disable the at least one branch to reduce overall power consumption.

8. The inference apparatus as described in claim 1, wherein the neural network model is configured to selectively clock-gated at least one of the plurality of branches to disable the at least one branch based on whether at least one condition is met, wherein the at least one condition includes one or a combination of bandwidth change and downsampling.

9. The inference device as described in claim 1, wherein the time offset value is greater than one.

10. A method for inferring from multiple input data using a neural network model involving at least hidden state inheritance, wherein multiple inference operations are performed on the multiple input data of the neural network model in parallel at multiple time steps, multiple hidden states corresponding to the multiple time steps are stored in a memory, and the method comprises: for each of the multiple inference operations: performing a feature extraction operation on one of the multiple input data to produce a extraction result; reading an inherited hidden state corresponding to a previous time step from the memory; performing a matrix operation based on the extraction result and the inherited hidden state to produce a current hidden state corresponding to a current time step, wherein the current hidden state is written into the memory for hidden state inheritance, a difference between the previous time step and the current time step is a time offset value, the time offset value being a time period count required to perform the matrix operation, and the time offset value being a positive integer; and performing an inference operation on one of a plurality of branches based on the current hidden state to produce an inference result.

Images