Gating Units for Gated Recurrent Neural Networks

Abstract

A method, computer program product, and computer system for gated recurrent neural networks (RNNs) are provided. The method may include: providing an element processor; providing different memory arrays for corresponding sets of one or more elements of a hidden state vector; storing column groups of a weight matrix into the memory arrays such that the set of one or more elements can be computed; computed one or more elements of each of the plurality of activation vectors using a set of one or more columns from the column groups associated with each of the plurality of activation vectors; and performing an element-wise gating operation on the computed elements by the element processor to produce a set of one or more elements.

Description

Technical Field

[0001] This invention relates to the field of digital computer systems, and more particularly, to gating units for gated recurrent neural networks. Background Technology

[0002] Neural networks are computational models used in artificial intelligence systems. A neural network is based on multiple artificial neurons. Each artificial neuron is connected to one or more other neurons, and these links can enhance or inhibit the activation state of neighboring neurons. The accuracy of neural networks on a variety of cognitive tasks has attracted considerable interest in designing hardware accelerators for training or inferring deep neural networks. In the training or inference of gated recurrent neural networks (RNNs) (such as Long Short-Term Memory (LSTM) networks), a set of element-wise operations is computed on a dimensional vector. However, these computations can be resource-intensive. Summary of the Invention

[0003] Exemplary embodiments disclose a method, computer program product, and computer system for a gated recurrent neural network (RNN). Exemplary embodiments may include: providing an element processor; providing different memory arrays for corresponding sets of one or more elements of a hidden state vector; storing columns of a set of weight matrices in the memory arrays such that the set of one or more elements can be computed; computed one or more elements of each of the plurality of activation vectors using a set of one or more columns from a column group associated with each of the plurality of activation vectors; and performing an element-wise gating operation on the computed elements by the element processor to produce a set of one or more elements. Attached Figure Description

[0004] The following detailed description, given by way of example and not intended to limit exemplary embodiments, will be better understood in conjunction with the accompanying drawings, in which:

[0005] Figure 1 A diagram depicting an electronic system according to an exemplary embodiment is shown.

[0006] Figure 2 A diagram depicting an electronic system according to an exemplary embodiment is shown.

[0007] Figure 3 A diagram depicting an electronic system according to an exemplary embodiment is shown.

[0008] Figure 4 A diagram depicting an electronic system according to an exemplary embodiment is shown.

[0009] Figure 5 A diagram depicting an electronic system according to an exemplary embodiment is shown.

[0010] Figure 6A A diagram depicting a memory array according to an exemplary embodiment is shown.

[0011] Figure 6B A diagram depicting a memory array according to an exemplary embodiment is shown.

[0012] Figure 6C A diagram depicting a computing memory according to an exemplary embodiment is shown.

[0013] Figure 6D A diagram depicting a computing memory according to an exemplary embodiment is shown.

[0014] Figure 7 A flowchart is depicted for a method for computing elements of a hidden state vector according to an exemplary embodiment.

[0015] Figure 8 An exemplary block diagram depicting hardware components according to an exemplary embodiment is shown.

[0016] Figure 9 A cloud computing environment according to an exemplary embodiment is described.

[0017] Figure 10 An abstract model layer according to an exemplary embodiment is described.

[0018] The accompanying drawings are not necessarily drawn to scale. They are schematic representations only and are not intended to depict specific parameters of exemplary embodiments. The drawings are intended to illustrate typical exemplary embodiments only. In the drawings, the same reference numerals denote the same elements. Detailed Implementation

[0019] The description of different embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0020] Recurrent neural networks (RNNs) can be artificial neural networks in which the connections between the network's nodes can form a graph (e.g., a directed graph) along a time series. This allows them to exhibit dynamic behavior over time. An RNN can be a network of neuron-like nodes organized into successive layers. Nodes can be either input nodes (receiving data from outside the network), output nodes (producing results), or hidden nodes (modifying data from the input to the output). Nodes in a RNN can include one or more gating units, and thus it can be called a gated recurrent neural network, where each node can include a gating unit. For example, gating units can include gated recurrent units (GRUs) or long short-term memory (LSTM) units.

[0021] The gating unit may include l G There are 10 gates. For example, in the case of a gated recursive unit, the number of gates is l. G It can be less than or equal to 3, l G ≤3, for example, for the smallest gate unit l G =2 and for fully gated unit l G =3. l G Each gate in the system can receive an input vector and, in response, provides a corresponding vector, which can be called an activation vector. The gated unit can perform a nonlinear transformation, which transforms the input signal x at time t... t And the hidden state h at the previous time step t-1 t-1 Mapped to the current hidden state h t Input signal x t and hidden state h t Each of these can be a vector. The gating unit can receive vector I at time t. t As input. For example, input vector I t It can be the input signal vector x at time t. t and the hidden state vector h at time t-1 t-1 The concatenation of the hidden state vector h. t It can be defined by the element-wise gating operation of the activation vector. G Each of the activation vectors can be defined as a function of the corresponding weight matrix W. For example, l G Each of the activation vectors can be a weighted vector P. j The function. The weighted vector P at time t. j It can be derived from the corresponding input vector I t and the different weight matrices W associated with the activation vector j It is produced by at least one multiplication.

[0022] Therefore, l G The weight matrix can be used to calculate the hidden state vector h according to this topic. tThe set of elements. The hidden state vector h t Can belong to a group Input signal x t Can belong to a group l G Each of the activation vectors can belong to a group. l G Each of the weight matrices can belong to a group. Where ss is the input vector I t The size of the input vector I at time t. For example, the input vector I at time t. t This can include the input signal vector x at time t. t The elements and the hidden state vector h at the previous time step t-1 t-1 The elements are such that s = k + d. The hidden state vector h... t It can include k elements. The hidden state vector h t Each element h t (i) (i varies between 1 and k) can have a corresponding element position, such as an index i.

[0023] Following the example of a fully gated unit, a gate can include a reset gate, an update gate, and an activation gate. The activation gate can provide a candidate hidden state vector. The activation vector of the reset gate can be defined as follows: r t =σ g (P r The activation vector of the update gate can be defined as follows: z t =σ g (P z The activation vector of an activation gate can be defined as follows: The function σ g This is the sigmoid function. It is the hyperbolic tangent function. P r P z and P h These are weighted vectors. For example, each weighted vector can be obtained as follows: and Among them W r W z and W h These are respectively related to the activation vector r t z t and The associated weight matrix. and These are the input vectors associated with the reset gate, update gate, and activation gate, respectively. Before inputting each input vector to the corresponding gate, the input vector can be... and Weighting is applied. Input vector It can be the input signal vector xt and hidden state vector h t-1 The concatenation of input vectors. It can be the input signal vector x t and hidden state vector h t-1 The concatenation of input vectors. It can be the input signal vector x t With the following vector r t ⊙h t-1 The concatenation of [variables]. In other words, the input vector associated with the gate can be modified by the gating unit, so that it is not only x t with h t-1 The series connection, but x t It can be obtained by a gate and a vector h. t-1 The element-wise operation on the output of the first vector produces another concatenation of vectors. In another example, each weighted vector can be obtained as follows: and Where b r b z and b h It can be a bias vector. In other words, the weighted vector is generated by multiplying the input vector by different weight matrices and adding the bias vector.

[0024] Therefore, training or inference of such gated recurrent neural networks can involve high-dimensional vectors (such as vectors) and A set of element-wise operations on a high-dimensional vector. However, computing element-wise operations on high-dimensional vectors in a hardware accelerator can be an expensive hardware implementation for the following reasons. First, scaling hardware implementations using vector sizes can be potentially expensive in terms of area and power requirements. Second, multiplexing can be difficult in resource-constrained applications because large vectors are stored in memory cells that are far apart from each other. This topic addresses this problem by mapping the parameters of a gated recurrent neural network to an array of computational cells that facilitates the design of a scalable and multiplexable hardware implementation of element-wise operations. The selective storage of columns of the weight matrix into corresponding locations in the memory array can be referred to as a mapping.

[0025] This topic enables efficient computation of the hidden state vector because it computes a set of one or more elements of the hidden state vector at a time. This contrasts with conventional methods, where the entire hidden state vector is computed only after each element of the activation vector has been computed. The set of elements (named SET) of the hidden state vector computed at once using this topic can contain n elements, where n is less than or equal to the number of elements in the hidden state vector (n ≤ k). In the case of n = k, this topic provides an alternative solution to the conventional method. For example, the number of elements n in the set can be a configurable parameter (e.g., the value n can be user-defined).

[0026] The elements of the hidden state vector can be divided into N distinct sets of elements: SET1, SET2, ..., SET3. N . Element sets SET1, SET2…SET N Each may or may not include the same number of elements, n. In one example, the number of elements, n, can be defined such that the ratio between the total number of elements, k, in the hidden state vector and the number of elements, n, in the set of elements, SET, is... It is an integer. For example, the element set SET1 can be labeled as h. t (i:j:l) indicates the hidden state vector h with positions / indices i, j, and l. t The elements are the elements of set SSET1.

[0027] This method can be executed to compute these sets of elements SET1, SET2...SET one at a time. N One approach is to compute all elements of the hidden state vector, repeating the method to compute the remaining uncomputed elements. For example, the method can be repeated N-1 times to compute the set of elements SET1, SET2…SSET. N In each iteration of this method, the element sets SSET1, SET2…SET can be computed one at a time. N A different set of elements in the hidden state vector. According to one embodiment, the method further includes repeated steps for one or more other different sets of elements in the hidden state vector until all elements of the hidden state vector have been computed. That is, for element sets SET1, SET2…SET… N Each element in the set SET j It can provide element processors (EP) j ) and memory array (MA j ), where j varies between 1 and N. For example, an electronic unit (e.g., an integrated circuit) comprising N element processors and N memory arrays can be provided, such that the element sets SET1, SET2…SET can be executed simultaneously or in parallel.N Calculation. Thus, according to one embodiment, the repetition of steps is performed simultaneously.

[0028] Alternatively, the element sets SET1, SET2…SET N Part of the calculation of can be performed sequentially, and another part of the calculation can be performed simultaneously. For this purpose, M element processors can be provided, where the number M of element processors is less than the number N of element sets, M < N. In this case, each element processor can be assigned a corresponding element set. In the case where an element processor is associated with two or more element sets, the element processor can be multiplexed across the inputs such that it calculates one element set at a time. However, in order to minimize the area requirements of the hardware implementation, the number M of element processors can be chosen to be as small as possible. Thus, according to one embodiment, the method further includes repeating the steps for one or more other different element sets of the hidden state vector until all elements of the hidden state vector have been calculated, where the (single) element processor is equipped with multiplexing features such that it calculates one element set at a time.

[0029] The element processor can calculate the corresponding element sets of the hidden state vector according to the following embodiments. Each element in the element set has a corresponding element position in the hidden state vector. The calculated elements of the activation vector respectively include multiple sets of one or more calculated elements of multiple activation vectors. Each set of the calculated elements has a set of element positions in the corresponding activation vector. For each activation vector among the multiple activation vectors, the calculation includes: receiving the values of the input vector at the memory array and respectively outputting a set of elements of the weighted vector (named result elements) from a set of one or more associated columns, the set of result elements having a set of element positions in the weighted vector; feeding the set of result elements to the element processor; and using the set of result elements to perform the calculation of the calculated element set of the activation vector.

[0030] According to one embodiment, the memory array includes row lines and column lines intersecting the row lines, and a computing memory coupled between the row lines and the column lines at the nodes formed by the row lines and the column lines. Each computing memory represents the value of the corresponding element in the column. The storage of the column includes storing all elements of each column in the computing memory of the corresponding single column line of the memory array.

[0031] According to one embodiment, the computing memory is any one of a resistive memory element, SRAM, or a flash memory element.

[0032] Figure 1 A diagram of an electronic system 100 according to an example of the present invention is depicted. The electronic system 100 can be a gated unit of a gated recurrent neural network. The gated unit can have a hidden state vector h tHidden state vector h t It can be made by l G activation vectors It is defined by element-wise gating operations. G Each of the activation vectors can be defined as the corresponding weight matrix W. i Functions. For example, l G Each of the activation vectors can be a weighted vector P. j The function. The weighted vector at time t can be obtained from the corresponding input vector I. t Multiplied by different weight matrices W associated with the activation vector j However, computing all elements of the hidden state vector at once can impose several limitations on hardware implementation. Electronic system 100 can solve this problem because it can be configured to compute the hidden state vector h. t A single element. The hidden state vector h t The element can be an element h with index i or element position i. t (i).

[0033] Electronic system 100 can be configured to perform a nonlinear transformation that transforms the input signal x at time t. t And the hidden state h at the previous time step t-1 t-1 Mapped to the current hidden state h t element h t (i). Hidden state vector h t Can belong to a group Input signal x t Can belong to a group l G Each of the activation vectors can belong to the group. l G Each of the weight matrices can belong to a group. Where s is the input vector I t The size of the input vector I at time t. For example, the input vector I at time t. t This can include the input signal vector x at time t. t The elements and the hidden state vector h at the previous time step t-1 t-1 The elements, therefore s = k + d. In another example, the input vector I associated with the gate. t It can be modified by the gating unit (before being input to the memory array), thus making it not only x t and h t-1 The series connection, but x t This is concatenated with another vector, which can be produced by element-wise operations on the output of a gate of element processor 103 and the vector. The hidden state vector h tIt can include k elements. The hidden state vector h t Each element h t (i) (where i varies between 1 and k) can have a corresponding element position, such as an index i. h t (1) is the hidden state vector h t The first element.

[0034] Electronic system 100 includes memory array 101 and element processor 103. Element processor 103 includes l G There are 105 gates, each providing an activation vector at time t. For example, in the case of the smallest gating unit l G =2, and in the case of LSTM l G =4. l G Each gate in gate 105 can have an activation function, such as the sigmoid function σ and the hyperbolic tangent function φ. The element processor 103 may also include other electronic components 106 that enable the element processor 103 to perform element-wise operations on vectors and apply activation functions. Figure 1 As indicated, electronic component 106 can be configured to perform element-wise multiplication, vector addition, and vector subtraction (vector addition with negative values).

[0035] Following the example of the smallest gating unit, the element processor 103 may include two gates, and the corresponding two activation vectors can be defined as follows:

[0036]

[0037] In this case, the hidden state vector can be defined as follows: The weighted vectors P1 and P2 can be equal to (at time t) w f x t +u f h t-1 +b y and w h x t +u h (f t ⊙h t-1 )+b h The weight matrix can be W. . and W h .

[0038] Memory array 101 includes features that enable it to store a column of data (named GRPs). i () multiple rows and columns. This column group includes elements that allow for the individual determination of activation vectors. elements weight matrix Columns. For example, elements The calculation may require an input vector I t And the column number i of the weight matrix W1. The column number (or position) i of the weight matrix W1 can include elements: W1(1,i), W2(2,i)...W1(s,i). Element V t 2 The calculation of (i) may require the input vector I. t And the number of columns i of the weight matrix W2. The number of columns i of the weight matrix W2 can include elements: W2(1, i), W2(2, i)...W2(s, i), etc. Therefore, it is possible to compute the element h. t (i) Column group GRP i This can include the weight matrix All columns 107 at position i. The memory array 101 can be configured to receive input vector I. t And output the corresponding weighted vector P at position 107 of each column. j The element P at position i j (i).

[0039] Following the example of the smallest gating unit, in order to compute element h t (i) may require calculating activation vector elements. and f t (i). To compute element f t (i) can be used to weigh the weight matrix W . The column at position i is stored in the column of memory array 101. To calculate the element... The weight matrix W can be h The column at position i is stored in another consecutive column of the memory array. Therefore, column group GRP i It consists of two columns.

[0040] The weighted vector P for each column in column 107 j element P j (i) The corresponding gates 105 of the element processor 103 are provided as input, such that the activation function of each gate in the gates 105 can be applied to the corresponding received weighted vector element P. i (i) on. This can generate activation vector elements. These activation vector elements It can be further processed by the element processor 103 using electronic components 106 in order to perform the operation that yields element h. t (i) element-wise operations.

[0041] Following the example of the smallest gating unit, element processor 103 can be configured to perform the following calculations: In this case, element-wise gating operations include element-wise multiplication.

[0042] Figure 2 A diagram illustrating an example of an electronic system 200 according to this subject is provided. The electronic system 200 can be configured to calculate the current hidden state h. t The two elements h t The set of (i1:i2). Electronic system 200 can be configured to perform a nonlinear transformation that transforms the input signal x at time t. t The hidden state h at the previous time step t-1 t-1 Mapped to the current hidden state h t The two elements h t The set of (i1:i2).

[0043] The electronic system 200 includes a memory array 201 and an element processor 203. The memory array 201 stores elements capable of determining activation vectors. elements weight matrix GRP column i1,i2 For example, element The calculation may require an input vector I t The weight matrix W1 has two columns at positions i1 and i2. The column at position i1 of the weight matrix W1 can include elements: W1(1,i1), W1(2,i1)...W1(s,i1). The column at position i2 of the weight matrix W1 can include elements: W1(1,i2), W1(2,i2)...W1(s,i2), and so on. Columns 207 of the same weight matrix are stored contiguously in a memory array. This minimizes the area requirements of the hardware implementation. The memory array 201 can be configured to receive the input vector I. t And output the corresponding weighted vector P at position 207 of each column. j The elements P at positions i1 and i2 j (i1:i2).

[0044] The weighted vector P for each column in column 207 j Each pair of elements P in j (i1:i2) are provided as input to the corresponding gates 205 of the element processor 203, such that the activation function of each gate in gate 105 can be applied to the corresponding received weighted vector element P. j (i1:i2) is used. This can generate activation vector elements. Element processor 203 can further process the activation vector elements using electronic component 206. In order to execute the operation that can obtain element h t Element-wise operations on the set (i1:i2).

[0045] Figure 3 A diagram illustrating an example of an electronic system 300 according to this subject is provided. The electronic system 300 can be configured to calculate the current hidden state h one by one. t All elements. Electronic system 300 can be configured to perform a nonlinear transformation that transforms the input signal x at time t. t The hidden state h at the previous time step t-1 t-1 Mapped to the current hidden state h t Therefore, the electronic system 300 may include a hidden state vector h. t Multiple element processors, ranging from 303.1 to 303.k, are used to handle the number of elements k, so that each element processor can provide a hidden state vector h. t The electronic system 300 also includes a memory array 301, which stores k column groups GRP1, GRP2...GRP k Store column groups GRP1, GRP2...GRP k The regions can be referred to as different memory regions (or different memory sub-regions). Each of the k column groups is stored contiguously. Each of the k column groups is capable of computing the hidden state vector h. t The corresponding columns. For example, column group GRP1 includes columns that enable the computation of the hidden state vector h. t element h t (1) Column. Each element processor in element processors 303.1 to 303.k can be configured to compute its hidden state vector h. t The corresponding elements, such as the reference Figure 1 As described in the element processor 103.

[0046] Figure 4 A diagram illustrating an example of an electronic system 400 according to this topic is provided. The electronic system 400 can be configured to calculate the current hidden state h one by one. t All elements. The electronic system 400 can be configured to perform a nonlinear transformation that transforms the input signal x at time t. t and the hidden state h at the previous time step t-1 t-1 Mapped to the current hidden state h t Therefore, the electronic system 400 may include a single-element processor 403. The memory array 401 of the electronic system 400 is similar to... Figure 3 The memory array 303 has k column groups GRP1, GRP2...GRP k(For example, k = 128) is stored. The electronic system 400 can be configured to multiplex the outputs of different groups using a single element processor 403. For example, as... Figure 4 As shown, a single element processor 403 is multiplexed across all groups to compute one hidden state vector element at a time. This implementation can be advantageous in resource-constrained applications.

[0047] Figure 4 An extension of the implementation can be to allow the use of more than one element processor that can be multiplexed across the desired set of groups. This is in Figure 5 As shown, for example, two element processors 503.1 and 503.2 are used. Electronic system 500 can be configured such that the output of one set of groups can be multiplexed using element processor 503.1, and the output of another set can be multiplexed using element processor 503.2. For example, as... Figure 5 As shown, element processor 503.1 is spanned across all groups GRP1 to GRP2. 60 Multiplexing for the hidden state vector h t The first 60 elements are computed one hidden state vector element at a time. Element processor 503.3 is used across all groups of GRP. 61 To GRP 128 Multiplexing for the hidden state vector h t The last 68 elements are used to calculate one hidden state vector element at a time.

[0048] Figure 6A A diagram illustrating a memory array 600 according to an example of this subject is provided. The memory array 600 can be referenced, for example, from... Figure 1 describe.

[0049] The memory array 600 may be a cross array of computing memory that provides local data storage and voltage sequences indicating the operation of the memory. Figure 6A This is a diagram of a two-dimensional (2D) cross array 600, which, for example, can perform matrix-vector multiplication. The cross array 600 consists of a set of conductive row wires 602.1…602.s and conductive column wires 608.1…608.1 that intersect with the set of conductive row wires 602.1…602.s. G form.

[0050] Conductive column wires can be called column lines, and conductive row wires can be called row lines. The number of rows and columns can be defined based on the number of elements in the hidden state vector to be computed. In this case, the first element of the hidden state vector can be computed. For this, it may be necessary to compute l g The first element of each activation vector in the activation vectors, therefore the memory array may include at least l GThe number of columns is defined by the number of rows in each weight matrix, i.e., s.

[0051] The intersection points between the sets of row wires and the sets of column wires are separated by computational memories, each memory having its own value, described as W(i,j), where i = 1…s and j = 1…i. G For ease of explanation, only one computational memory 620 is used. Figure 6A The elements are marked with reference numbers. For example, column 608.1 of the memory array stores the element of the column at position 1 of the weight matrix W1. Column 608.2 of the memory array stores the element of the column at position 1 of the weight matrix W2, and so on.

[0052] Input voltage v1…v s These are applied to row conductors 602.1-602.s respectively. Each column conductor 608.1-608.k corresponds to the current generated by each computed memory along a specific column conductor. Summation. For example, as... Figure 6A As shown, the current I2 generated by the conductor 608.2 is calculated according to the equation I2=v1·W2(1,1)+v2·W2(2,2)+v3·W2(3,1)+…+v s ·W2(s,1). Therefore, array 600 multiplies the value stored in computation memory 620 by the voltage v1-v s The defined row wires are used to compute matrix-vector multiplication. Therefore, the memory array itself, along with the relevant row or column wires of array 600, can be used to perform multiplication locally at each compute memory 620 of array 600.

[0053] Figure 6B A diagram illustrating a memory array 700 according to an example of this subject is provided. For example, memory array 700 can be referenced... Figure 1 The description is as follows. Memory array 700 can be a cross-array of memristor devices. For a set of s distinct parallel lines arranged horizontally (s rows) and perpendicular to the s lines (l... G 2l (column) arrangement G Another set of distinct parallel lines, such that in all sx2l of these lines... G At each intersection point, a memristor device exists, and the memory array 700 forms the intersections of the PCM devices. The current through a single memristor device is the product of the voltage applied to it and its conductance. Since the conductance is always positive, a single column consists of two vertical lines that together represent the signed values ​​in the array, and the net current of that column is obtained by subtracting the currents of these two lines, as given in the following equation. in, as well as W 9The positive portion of the weights can be encoded, and W - It can encode the negative part of the weights. For example, if the weight matrix W is positive, W... + =W and W - =0, and if W is negative, W + =0 and W - =|W|.

[0054] Matrix-vector multiplication can be performed by applying voltages along the rows of the cross and summing the currents along the columns in the cross (using Kirchhoff's current law). The net output current is typically measured using an analog-to-digital converter (ADC) for further calculations. An ADC can convert analog values ​​to their digital representations as long as the input analog values ​​are always bounded; otherwise, the ADC output saturates. In another example, each cross array can use one or fewer ADCs to sample the column currents. The output of any group can be constrained to the interval [-4, 4]. This facilitates the use of one or fewer ADCs that can be multiplexed across all columns.

[0055] Another example of a memory array could be a digital systolic array of computational memories. A systolic array can be a homogeneous array of tightly coupled computational memories. Each computational memory has a memory for storing parameter values ​​and typically performs logical operations such as multiplication and addition. In a systolic array, input data can be read only once and reused multiple times without being stored back into any intermediate registers. Only adjacent data processing units within the systolic array can be wire-coupled, resulting in short and energy-efficient wiring. This implementation accelerates matrix multiplication compared to GPUs / CPUs. Specifically, systolic arrays can be used to implement gated RNNs. In such an implementation, this mapping technique can be used, making the design of gating operation computational units scalable and multiplexable.

[0056] Figure 6C An example of a computational memory 800 according to this subject is depicted. The computational memory 800 may be an SRAM cell. An SRAM cell is an electronic circuit capable of storing 1 bit of memory. The ability to change the memory state of an SRAM cell makes it a suitable candidate for computational memory. Figure 6C A 6-transistor SRAM cell is shown.

[0057] Figure 6D An example computational memory 810 according to this topic is depicted. The computational memory 810 can be flash memory. A flash memory device is an electronic device capable of storing 1 bit of memory. The ability of a flash memory device to change the memory state makes it a suitable candidate for computational memory.

[0058] Figure 7This is a flowchart of a method for computing a set of one or more elements of the hidden state vector of a gated recurrent neural network. The gated RNN includes at least one gating unit, as shown in the reference diagram. Figures 1-6A ~ Figure 6D Described.

[0059] The term "set of elements" refers to a specific number of elements of a vector having corresponding positions in the vector. For simplicity, let's assume the set of elements includes three elements, for example, elements with positions 1, 2, and 3 respectively. In this case, the set of elements refers to the three elements with positions 1, 2, and 3; for example, stating "the set of elements of vector X" means the three elements in vector X with positions 1, 2, and 3. That is, this method enables the computation of the hidden state vector h. t element h t (1) h t (2) and h t (3) (marked as h) t (1:2:3)). To simplify the description, we further assume that the gated unit is an LSTM unit. That is, the hidden state vector h can be defined as follows: t .

[0060] f t =σ g (w f x t +u f h t-1 +b f )=σ g (W f I t +b f (Equation 1)

[0061] i t =σ g (w i x t +u i h t-1 +b)=σ g (W i I t +b i (Equation 2)

[0062] o t =σ g (w o x t +u o h t-1 +b o )=σ g (W o I t +bo (Equation 3)

[0063]

[0064] In step 901, it becomes possible to compute the set of elements h. t The column groups of the (1:2:3) weight matrix can be stored in a memory array. To compute the set of elements, the corresponding elements of the activation vectors defined by Equations 1 through 4 may be required. That is, the activation vector elements o can be computed. t (1:2:3),f t (1:2:3),i t (1:2:3) and In order to calculate the element h t (1:2:3). For example, to calculate element f t (1:2:3), the weight matrix W can be used. . The subgroup sb1 of the column at positions 1, 2, and 3. To calculate element i... t (1:2:3), the weight matrix W can be used. i The subgroup sb2 of the columns at positions 1, 2, and 3. To calculate element o t (1:2:3), the weight matrix W can be used. o The subgroup sb3 of columns at positions 1, 2, and 3. To calculate the elements... The weight matrix W can be used c The column group consists of subgroups sb4 at positions 1, 2, and 3. Therefore, the column group can be formed by subgroups sb1, sb2, ab3, and sb4. This column group may include the weight matrix W. . The three columns and weight matrix W i The three columns and weight matrix W o The three columns and the weight matrix W c The three columns. Each column in this group can be stored in the corresponding column of the memory array. Columns of each subgroup in subgroups sb1, sb2, ab3, and sb4 can be stored consecutively. Using... Figure 6A The cross-array can be used to store subgroups sb1 in consecutive columns 608.1, 608.2, and 608.3. Subgroups sb2 can be stored in consecutive columns 608.4, 608.5, and 608.6. Subgroups sb3 can be stored in consecutive columns 608.7, 608.8, and 608.9. Subgroups sb4 can be stored in consecutive columns 608.10, 608.11, and 608.12.

[0065] In step 903, the corresponding subgroups sb1, sb2, ab3, and sb4 of the columns in the subgroup of columns can be used to compute the set of elements of each activation vector in the activation vector. For example, element f can be computed using subgroup sb1 according to Equation 1. t (1:2:3). The element i can be calculated using subgroup sb2 according to equation 2. t (1:2:3). The element o can be calculated using subgroup sb3 according to equation 3. t (1:2:3). The elements can be calculated using subgroup sb4 according to Equation 4.

[0066] In step 905, the element processor can calculate the element f according to equations 5 and 6. t (1:2:3),i t (1:2:3),o t (1:2:3) and Perform an element-wise gating operation on the top. This produces a set of elements h. t (1:2:3). In this example, element-wise gating involves element-wise multiplication and the application of an activation function.

[0067] Figure 8 A block diagram of a hardware device according to an exemplary embodiment is depicted. It should be understood that... Figure 8 This is merely an illustration of an implementation and does not imply any limitation regarding the environment in which different embodiments may be implemented. Many modifications can be made to the depicted environment.

[0068] The devices used herein may include one or more processors 02, one or more computer-readable RAMs 04, one or more computer-readable ROMs 06, one or more computer-readable storage media 08, device drivers 12, read / write drivers or interfaces 14, and network adapters or interfaces 16, all interconnected via a communication structure 18. The communication structure 18 may be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communication and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system.

[0069] One or more operating systems 10 and one or more application programs 11 are stored on one or more computer-readable storage media 08 for execution by one or more processors 02 via one or more corresponding RAMs 04 (which typically include cache memory). In the illustrated embodiment, each computer-readable storage medium 08 may be a disk storage device such as an internal hard disk, CD-ROM, DVD, Memory Stick, magnetic tape, magnetic disk, optical disk, semiconductor storage device such as RAM, ROM, EPROM, flash memory, or any other computer-readable tangible storage device capable of storing computer programs and digital information.

[0070] The device used herein may also include a R / W driver or interface 14 for reading from and writing to one or more portable computer-readable storage media 26. An application program 11 on the device may be stored on one or more portable computer-readable storage media 26, read from and loaded onto a corresponding computer-readable storage medium 26 via the appropriate R / W driver or interface 14.

[0071] The device used herein may also include a network adapter or interface 16, such as a TCP / IP adapter card or a wireless communication adapter (such as a 4G wireless communication adapter using OFDMA technology). The application 11 on the computing device may be downloaded to the computing device from an external computer or external storage device via a network (e.g., the Internet, a local area network, or other wide area networks or wireless networks) and the network adapter or interface 16. The program may be loaded from the network adapter or interface 16 onto a computer-readable storage medium 08. The network may include copper wires, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers.

[0072] The device used herein may also include a display screen 20, a keyboard or keypad 22, and a computer mouse or touchpad 24. The device driver 12 is connected to the display screen 20 for imaging, the keyboard or keypad 22, the computer mouse or touchpad 24, and / or the display screen 20 for pressure sensing for alphanumeric character input and user selection. The device driver 12, R / W driver or interface 14, and network adapter or interface 16 may include hardware and software (stored on computer-readable storage medium 08 and / or ROM 06).

[0073] The procedures described herein are identified based on their implementation in a specific exemplary embodiment. However, it should be understood that any particular procedural terminology used herein is merely for convenience, and therefore the exemplary embodiments should not be limited to use only in any specific application identified and / or implied by such terminology.

[0074] Based on the foregoing, a computer system, method, and computer program product have been disclosed. However, various modifications and substitutions can be made without departing from the scope of the exemplary embodiments. Therefore, exemplary embodiments have been disclosed by way of example rather than limitation.

[0075] It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings set forth herein is not limited to cloud computing environments. Rather, exemplary embodiments can be implemented in conjunction with any other type of computing environment now known or developed hereafter.

[0076] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0077] The features are as follows:

[0078] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing functions, such as server time and network storage, as needed, without requiring manual interaction with the service provider.

[0079] Wide Area Network Access: The functionality is available over a network and accessed through a standard mechanism that facilitates the use of heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0080] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location independence because consumers typically do not control or know the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0081] Rapid and flexible: In some cases, this means being able to quickly and flexibly provide functionality to automatically and rapidly scale outwards and rapidly release it to scale inwards. For consumers, the functionality available for supply often appears unrestricted and can be purchased at any time and in any quality.

[0082] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both the providers and consumers of the services being utilized.

[0083] The service model is as follows:

[0084] Software as a Service (SaaS): The functionality provided to consumers is the use of the provider's applications running on cloud infrastructure. These applications can be accessed from various client devices through a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application functionality, with the possible exception of limited user-specific application configuration settings.

[0085] Platform as a Service (PaaS): Provides consumers with the ability to deploy consumer-created or acquired applications onto cloud infrastructure using programming languages ​​and tools supported by the provider. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environments.

[0086] Infrastructure as a Service (IaaS): This provides consumers with the processing, storage, networking, and other basic computing resources to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0087] The deployment model is as follows:

[0088] Private cloud: Cloud infrastructure operated solely by an organization. It can be managed by the organization or a third party, and can exist internally or externally.

[0089] Community cloud: Cloud infrastructure shared by several organizations to support specific communities with common concerns (e.g., tasks, security requests, policies, and compliance considerations). It can be managed by an organization or a third party and can exist internally or externally.

[0090] Public cloud: Cloud infrastructure available to the public or large industrial groups and owned by organizations that sell cloud services.

[0091] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0092] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure of a network of interconnected nodes.

[0093] Now for reference Figure 9 The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 40 and local computing devices used by cloud consumers: such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptops 54C, and / or automotive computer systems 54N. The nodes 40 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 9 The types of computing devices 54A-N shown are for illustrative purposes only, and computing node 40 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0094] Now for reference Figure 10 This demonstrates a cloud computing environment of 50 ( Figure 9 This provides a set of functional abstraction layers. It should be understood beforehand that... Figure 10 The components, layers, and functions shown are for illustrative purposes only, and exemplary embodiments are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0095] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a host 61; a server 62 based on a RISC (Reduced Instruction Set Computer) architecture; a server 63; a blade server 64; a storage device 65; and networking and interconnection components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0096] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage 72; virtual network 73, including virtual private network; virtual application and operating system 74; and virtual client 75.

[0097] In one example, management layer 80 may provide the following functionalities: Resource Provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost tracking when utilizing resources within the cloud computing environment and issues invoices or bills for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 85 provides pre-scheduling and procurement of cloud resources, whereby future demand for cloud resources is anticipated according to the SLA.

[0098] The workload layer 90 provides examples of functionalities that can be leveraged in a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analytics and processing 94; transaction processing 95; and neural network computing 96.

[0099] Exemplary embodiments may be systems, methods, and / or computer program products at any possible level of integration technical detail. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the exemplary embodiments.

[0100] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0101] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0102] Computer-readable program instructions used to perform operations of exemplary embodiments may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., using an internet service provider via the internet). In some embodiments, to perform aspects of exemplary embodiments, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing state information from the computer-readable program instructions.

[0103] This document describes aspects of exemplary embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to exemplary embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0104] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of manufacture comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0105] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various exemplary embodiments. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or performs a combination of dedicated hardware and computer instructions.

Claims

1. A method for a gated recurrent neural network (RNN) having a hidden state vector, wherein the hidden state vector is defined by an element-wise gating operation of a plurality of activation vectors, wherein each activation vector of the plurality of activation vectors is a function of a weighted vector, and wherein the weighted vector is produced by an input vector multiplied by a different weight matrix associated with the activation vector, the method comprising: providing an element processor; providing different memory arrays for respective sets of one or more elements of the hidden state vector; storing groups of columns of weight matrices in the memory arrays such that the sets of one or more elements can be computed; computing one or more elements of each of the plurality of activation vectors using a set of one or more columns of the groups of columns associated with each of the plurality of activation vectors; and performing the element-wise gating operation by the element processor on the computed elements, thereby producing the sets of one or more elements.

2. The method of claim 1, further comprising: providing different other memory arrays for respective other sets of one or more elements of the hidden state vector; storing other groups of columns of weight matrices in the other memory arrays such that the other sets of one or more elements can be computed; computing one or more elements of each of the plurality of activation vectors using a further set of one or more columns of the other groups of columns associated with each of the plurality of activation vectors; and performing the element-wise gating operation by the element processor on the computed elements, thereby producing the other sets of one or more elements.

3. The method of claim 2, wherein: the storing of the other groups of columns, the computing of the one or more elements of each of the plurality of activation vectors, and the performing of the element-wise gating operation by the element processor on the computed elements are performed simultaneously.

4. The method of claim 2, wherein: the providing of the different other memory arrays, the storing of the other groups of columns, the computing of the one or more elements of each of the plurality of activation vectors, and the performing of the element-wise gating operation by the element processor on the computed elements are repeatedly performed until all elements of the hidden state vector are computed; and the element processor is equipped with a multiplexing feature, thereby causing the element processor to compute one set of elements at a time.

5. The method of claim 1, wherein: each element of the sets of one or more elements has a respective element position in the hidden state vector; the computed one or more elements include a plurality of sets of one or more computed elements of the plurality of activation vectors, respectively; each set of one or more computed elements includes a set of one or more element positions in a respective activation vector; and computing one or more elements for each of the activation vectors comprises: receiving values of the input vector at the memory array and outputting a set of result elements of the weighted vector from an associated set of one or more columns, respectively, wherein the set of result elements comprises the set of one or more element positions in the weighted vector; feeding the set of result elements to the element processor; and performing the computation of the set of one or more computed elements of the activation vector using the set of result elements.

6. The method of claim 1, wherein: the memory array comprises row lines, column lines intersecting the row lines, and compute memories coupled between the row lines and the column lines at junctions formed by the row lines and the column lines; a compute memory of the compute memories represents values of elements of one or more columns of the column group; and storing the column group comprises storing all elements of each column of the column group in compute memories of a respective single column line of the memory array.

7. The method of claim 6, wherein, the compute memories are any of resistive memory elements, static random access memory (SRAM) elements, or flash elements.

8. A computer program product for a gated recurrent neural network (RNN) having a hidden state vector, wherein, the hidden state vector is defined by an element-wise gating operation of a plurality of activation vectors, wherein each activation vector of the plurality of activation vectors is a function of a weighted vector, and wherein the weighted vector results from an input vector multiplied by a different weight matrix associated with the activation vector, the computer program product comprising: one or more non-transitory computer-readable storage media and program instructions stored on the one or more non-transitory computer-readable storage media capable of executing a method comprising: providing an element processor; providing a different memory array for a respective set of one or more elements of the hidden state vector; storing a column group of weight matrices in the memory array such that the set of one or more elements can be computed; computing one or more elements of each of the plurality of activation vectors using a set of one or more columns of the column group associated with each of the plurality of activation vectors; and performing the element-wise gating operation on the computed elements by the element processor resulting in the set of one or more elements.

9. The computer program product of claim 8, further comprising: providing a different other memory array for a respective other set of one or more elements of the hidden state vector; storing a further column group of weight matrices in the other memory array such that the other set of one or more elements can be computed; computing one or more elements of each of the plurality of activation vectors using a further set of one or more columns of the further column group associated with each of the plurality of activation vectors; and performing the element-wise gating operation on the computed elements by the element processor resulting in the other set of one or more elements.

10. The computer program product of claim 9, wherein: ​ storing the other column groups, computing the one or more elements of each of the plurality of activation vectors, and performing the element-wise gating operation on the computed elements by the element processor are performed concurrently.

11. The computer program product of claim 9, wherein: providing the different other memory array, storing the other column groups, computing the one or more elements of each of the plurality of activation vectors, and performing the element-wise gating operation on the computed elements by the element processor are repeatedly performed until all elements of the hidden state vector are computed; and the element processor is equipped with a multiplexing feature such that the element processor computes one set of elements at a time.

12. The computer program product of claim 8, wherein: each element in the set of one or more elements has a respective element position in the hidden state vector; the computed one or more elements comprise a plurality of sets of one or more computed elements of the plurality of activation vectors, respectively; each set of one or more computed elements comprises a set of one or more element positions in a respective activation vector; and computing one or more elements for each of the activation vectors comprises: receiving values of the input vector at the memory array and outputting a resulting set of elements of the weighted vector from an associated set of one or more columns, respectively, wherein the resulting set of elements comprises a set of one or more element positions in the weighted vector; feeding the resulting set of elements to the element processor; and performing the computation of the set of one or more computed elements of the activation vector using the resulting set of elements.

13. The computer program product of claim 8, wherein: the memory array comprises row lines, column lines intersecting the row lines, and computational memories coupled between the row lines and the column lines at junctions formed by the row lines and the column lines; a computational memory in the computational memories represents a value of an element of one or more columns in the column groups; and storing the column groups comprises storing all elements of each column in the column groups in a computational memory of a respective single column line of the memory array.

14. The computer program product of claim 13, wherein, the computational memories are any of resistive memory elements, static random access memory (SRAM) elements, or flash memory elements.

15. A computer system for a gated recurrent neural network (RNN) with a hidden state vector, wherein, the hidden state vector is defined by an element-wise gating operation of a plurality of activation vectors, wherein each activation vector in the plurality of activation vectors is a function of a weighted vector, and wherein the weighted vector results from an input vector multiplied by a different weight matrix associated with the activation vector, the computer system comprising: one or more computer processors, one or more computer-readable storage media, and program instructions stored on the one or more of the computer-readable storage media for execution by at least one of the one or more processors capable of performing a method comprising: providing an element processor; providing a different memory array for a respective set of one or more elements of the hidden state vector; storing a group of columns of a weight matrix in the memory array such that the set of one or more elements can be computed; computing one or more elements of each of the plurality of activation vectors using a set of one or more columns of the group of columns associated with each of the plurality of activation vectors; and performing the element-wise gating operation on the computed elements by the element processor resulting in the set of one or more elements.

16. The computer system of claim 15, further comprising: providing a different other memory array for a respective other set of one or more elements of the hidden state vector; storing a further group of columns of a weight matrix in the other memory array such that a further set of one or more elements can be computed; computing one or more elements of each of the plurality of activation vectors using a further set of one or more columns of the further group of columns associated with each of the plurality of activation vectors; and performing the element-wise gating operation on the computed elements by the element processor resulting in the further set of one or more elements.

17. The computer system of claim 16, wherein: storing the further group of columns, computing the one or more elements of each of the plurality of activation vectors, and performing the element-wise gating operation on the computed elements by the element processor are performed simultaneously.

18. The computer system of claim 16, wherein: providing the different other memory array, storing the further group of columns, computing the one or more elements of each of the plurality of activation vectors, and performing the element-wise gating operation on the computed elements by the element processor are repeatedly performed until all elements of the hidden state vector are computed; and the element processor is equipped with a multiplexing feature such that the element processor computes one set of elements at a time.

19. The computer system of claim 15, wherein: each element of the set of one or more elements has a respective element position in the hidden state vector; the computed one or more elements include a plurality of sets of one or more computed elements of the plurality of activation vectors, respectively; each set of one or more computed elements includes a set of one or more element positions in a respective activation vector; and computing one or more elements for each of the activation vectors includes: receiving values of the input vector at the memory array and outputting a resulting set of elements of the weighted vector from the associated set of one or more columns, respectively, wherein the resulting set of elements includes the set of one or more element positions in the weighted vector; feeding the resulting set of elements to the element processor; and performing the computation of the set of one or more computed elements of the activation vector using the resulting set of elements.

20. The computer system of claim 15, wherein: ​ The memory array includes row lines, column lines intersecting the row lines, and compute memories coupled between the row lines and the column lines at junctions formed by the row lines and the column lines; A compute memory of the compute memories represents values of elements of one or more columns of the column group; And Storing the column group includes storing all elements of each column of the column group in compute memories of a respective single column line of the memory array.

Images