RNN-based operator computation method, apparatus, device, computer program product, and medium

Abstract

An RNN operator computing method, device, equipment, computer program product and medium are provided. The method is for an RNN operator. First, sequence data containing a plurality of time step input matrices is acquired, and the input matrices are subjected to single matrix-matrix multiplication operation with a layer input weight matrix, thereby generating, at one time, intermediate result matrices corresponding to all time steps for gated calculation. Subsequently, based on a binary semaphore mechanism, the production completion state of the batch of intermediate result matrices is synchronized among a plurality of parallel computing units of a chip. Once the production is confirmed to be completed, each computing unit performs iterative update calculation of the hidden space state in time step order using the ready intermediate result matrices. In the iteration process, the production state of the latest hidden space state is also synchronized between adjacent time step computing units through the binary semaphore mechanism, to ensure the correctness of the time sequence dependency, and finally the hidden space state output corresponding to the entire input sequence is obtained.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and more particularly, to a recurrent neural network (RNN) class operator computing method, apparatus, device, computer program product and storage medium. BACKGROUND

[0002] Recurrent neural networks and their derivative architectures, such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), are the core operators in the field of deep learning for processing sequence modeling tasks. Due to their recurrent connection characteristics, these operators can capture the temporal dependence and dynamic context information in the data, and are widely used in many key fields such as speech recognition, natural language processing, machine translation, time series prediction, etc.

[0003] With the industrialization of artificial intelligence technology, the requirements for the inference performance of the model are increasingly stringent, especially in real-time or near real-time application scenarios. Therefore, when deployed on special artificial intelligence chips (such as GPU, TPU, etc.) for inference, the implementation of RNN class operators not only requires accurate calculation results, but also requires low latency and high hardware resource utilization in the calculation process. Low latency is the basis for ensuring user experience and system response speed, while high hardware resource utilization is the key to maximizing the peak computing power of expensive acceleration chips and reducing the unit computing cost. However, the traditional implementation of RNN class operators often has difficulty in effectively organizing computation and data flow to fully utilize the parallel computing units and high-bandwidth memory subsystems provided by the chip.

[0004] Therefore, there is a need for an optimization method for efficient inference of RNN class operators, which can significantly improve the calculation speed and hardware utilization efficiency on artificial intelligence chips while maintaining model accuracy. SUMMARY

[0005] To solve the above problems, the present disclosure converts the traditional calculation executed in time steps in series into an efficient mode more suitable for parallel hardware architecture through algorithm reconstruction and hardware co-design.

[0006] Embodiments of the present disclosure provide a recurrent neural network class operator computing method, apparatus, device, computer program product and computer readable storage medium.

[0007] Embodiments of the present disclosure provide a recurrent neural network type operator calculation method, comprising: obtaining input sequence data, the input sequence data comprising input matrices of a first number of time steps; performing matrix multiplication operation on a matrix composed of the input matrices of the first number of time steps and a layer input weight matrix to generate a first number of intermediate result matrices corresponding to the first number of time steps and related to gated calculation; synchronizing production states of the first number of intermediate result matrices between a plurality of parallel computing units of an artificial intelligence chip based on a binary semaphore mechanism; in response to the production states of the first number of intermediate result matrices being a completed state, performing iterative update calculation of hidden space states on the plurality of parallel computing units using the corresponding intermediate result matrices in time step order based on the binary semaphore mechanism to obtain a final hidden space state output corresponding to the input sequence data, wherein the production states of the latest hidden space states are synchronized between the parallel computing units for calculation of adjacent time steps based on the binary semaphore mechanism.

[0008] Embodiments of the present disclosure provide a recurrent neural network type operator calculation device, comprising: a layer input obtaining module configured to obtain input sequence data, the input sequence data comprising input matrices of a first number of time steps; a layer input mapping module configured to perform matrix multiplication operation on a matrix composed of the input matrices of the first number of time steps and a layer input weight matrix to generate a first number of intermediate result matrices corresponding to the first number of time steps and related to gated calculation; a layer input mapping synchronization module configured to synchronize production states of the first number of intermediate result matrices between a plurality of parallel computing units of an artificial intelligence chip based on a binary semaphore mechanism; and a hidden space state updating module configured to, in response to the production states of the first number of intermediate result matrices being a completed state, perform iterative update calculation of hidden space states on the plurality of parallel computing units using the corresponding intermediate result matrices in time step order based on the binary semaphore mechanism to obtain a final hidden space state output corresponding to the input sequence data, wherein the production states of the latest hidden space states are synchronized between the parallel computing units for calculation of adjacent time steps based on the binary semaphore mechanism.

[0009] Embodiments of the present disclosure provide a recurrent neural network type operator calculation device, comprising: one or more processors; and one or more memories having computer executable instructions stored therein, when the computer executable instructions are executed by the processors, performing the recurrent neural network type operator calculation method as described above.

[0010] The embodiment of the present disclosure provides a computer readable storage medium, which has stored computer executable instructions, and the computer executable instructions are used to implement the recurrent neural network type operator calculation method as described above when executed by a processor.

[0011] The embodiment of the present disclosure provides a computer program product, which is stored on a computer readable storage medium and includes computer executable instructions, and the computer executable instructions make a data processing device execute the recurrent neural network type operator calculation method according to the embodiment of the present disclosure when executed by a processor.

[0012] The method provided by the embodiment of the present disclosure is aimed at the RNN type operator on an artificial intelligence chip. First, sequence data containing a plurality of time step input matrices are acquired, and the input matrices as a whole are subjected to single matrix-matrix multiplication operation with a layer input weight matrix, so as to generate, at one time, intermediate result matrices corresponding to all time steps for gated calculation. Subsequently, based on a binary semaphore mechanism, the production completion state of the batch of intermediate result matrices is synchronized among a plurality of parallel computing units of the chip. Once the production is confirmed to be completed, each computing unit performs iterative update calculation of the hidden space state in the time step sequence by using the ready intermediate result matrices, wherein in the iteration process, the production state of the latest hidden space state is also synchronized between the computing units of adjacent time steps by the binary semaphore mechanism, so as to ensure the correctness of the time sequence dependence, and finally the hidden space state output corresponding to the entire input sequence is obtained.

[0013] The method of the embodiment of the present disclosure can reconstruct the time step serial matrix-vector multiplication (GEMV) operation which is traditionally difficult to parallelize due to time sequence dependence into large-scale matrix-matrix multiplication (GEMM) operation which can be executed in parallel at one time, greatly improving the calculation density, and thus fully exerting the hardware peak computing power of the artificial intelligence chip. Secondly, by introducing the binary semaphore mechanism for synchronization within a single kernel, the data dependence between the layer input mapping calculation and the iterative update calculation of the hidden space state, and between the state update of adjacent time steps is effectively managed, which not only ensures the strict time sequence requirement of the RNN iterative calculation, but also enables the state update of different time steps to be executed in a pipelined manner on a plurality of parallel computing units, realizing parallelization in the time dimension, and completely avoiding the huge scheduling and synchronization overhead caused by multiple start of independent kernels in the traditional scheme. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the description of the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some of the example embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0015] Figure 1 is a schematic flow chart illustrating a conventional implementation of an LSTM operator;

[0016] Figure 2 is a flow chart illustrating a recurrent neural network-like operator computation method according to an embodiment of the present disclosure;

[0017] Figure 3 is a schematic flow chart illustrating an implementation of an LSTM operator according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram illustrating a recurrent neural network-like operator computation apparatus according to an embodiment of the present disclosure; and

[0019] Figure 5 is a schematic diagram illustrating a recurrent neural network-like operator computation apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] In order to make the objectives, technical solutions and advantages of the present disclosure more obvious, the example embodiments according to the present disclosure will be described in detail below with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the example embodiments described herein.

[0021] In the present specification and drawings, the same or similar reference signs are used to represent steps and elements having substantially the same or similar functions, and repeated descriptions of these steps and elements will be omitted. Meanwhile, in the description of the present disclosure, the terms “first”, “second”, and the like are only used to distinguish the description, and cannot be understood as indicating or implying relative importance or sequence.

[0022] In the embodiments of the present disclosure, the term “module” or “unit” refers to a computer program or a part of a computer program having a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that contains the function of the module or unit.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0024] For the convenience of describing the present disclosure, the following introduces the concepts related to the present disclosure.

[0025] The recurrent neural network type operator calculation method of the present disclosure can be based on artificial intelligence (AI). Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence.

[0026] The recurrent neural network type operator calculation method of the present disclosure can be based on an artificial intelligence chip. Artificial intelligence chips are a class of hardware processors specially designed for efficient execution of artificial intelligence computing tasks. Unlike general-purpose processors (such as CPUs), they usually adopt parallel computing architecture, integrate a large number of computing cores optimized for matrix / tensor operations, and aim to provide extremely high computing throughput and energy efficiency to meet the needs of compute-intensive applications such as deep learning. Typical representatives of such chips include graphics processing units (GPUs), tensor processing units (TPUs), and various neural network processors (NPUs).

[0027] The recurrent neural network type operator calculation method of the present disclosure can be based on a recurrent neural network (RNN) operator. RNN operators are a class of artificial intelligence model core components for processing sequence data, such as LSTM and GRU, whose core mathematical features are characterized by the introduction of "hidden space state", which is passed between time steps, so that the output calculation at the current time depends not only on the current input, but also on the "memory" of all past time steps. This structure gives RNN the ability to capture temporal dependencies and contextual information, but also causes its calculation process to have strict sequential dependencies in the time dimension.

[0028] The recurrent neural network type operator computing method of the present disclosure can also be based on a binary semaphore mechanism. A binary semaphore is a synchronization primitive used in parallel computing to coordinate access to shared resources or ensure the execution order between tasks among multiple execution units (such as threads, computing cores), whose core is a variable that can only take two states (such as "0" and "1"), and is matched with indivisible atomic operations. The typical working mode of the binary semaphore mechanism includes that a producer unit sets the semaphore to a "ready" state after completing data production and ensuring data visibility; a consumer unit safely reads and uses the data by waiting for the semaphore to become "ready", thereby avoiding data competition or using unready data.

[0029] The recurrent neural network type operator computing method of the present disclosure can also be based on a domain-specific architecture (DSA). As a processor design paradigm, the core idea of the DSA is to customize computing units, memory levels and interconnection structures according to the common computing mode and data flow characteristics of a specific application field (such as graphics rendering, deep learning), so as to achieve performance and energy efficiency far exceeding that of general-purpose architectures in this field. For example, the DSA for deep learning usually contains highly optimized matrix multiplication units, and may use reduced numerical calculation precision to pursue extreme performance.

[0030] In summary, the scheme provided by the embodiments of the present disclosure relates to the technologies of artificial intelligence, artificial intelligence chips, RNN operators, binary semaphore mechanisms, DSAs and the like, and the embodiments of the present disclosure will be further described below in conjunction with the accompanying drawings.

[0031] In the following description of the present disclosure, the LSTM operator is described as an example of the RNN type operator, but it should be understood that the method of the present disclosure can be widely applied to various RNN type operators, rather than being limited to only the LSTM operator.

[0032] Figure 1 is a schematic flowchart showing a traditional implementation of the LSTM operator.

[0033] As Figure 1As shown, it shows the traditional implementation of LSTM operator, i.e., a serial computation process strictly unfolded along the time axis, where each computation unit (i.e., RNN cell) corresponds to the processing of one time step. Specifically, the computation of LSTM operator starts from the layer input data, which is sliced into consecutive segments along the time dimension. At each time step, the input matrix of current time is first multiplied with the weight matrix Wi, combined with the bias value, while the "hidden space state" passed from the last time step is multiplied with another set of weight matrix Wh. The results of these two parts of linear transformation are added together, and then passed through an activation function (e.g., Sigmoid or Tanh, etc.), to generate the input gate, forget gate, cell candidate value and output gate of current time. Subsequently, these gate signals interact with each other according to the mathematical definition of LSTM, to update the cell state and produce the new hidden space state of current time. This new state is then passed to the next time step as the starting point of its computation, and at the same time, it can be used as part of the layer output activation of the current time step. This process is repeated until the end of the sequence, and finally produces the "final hidden space state output".

[0034] For example, the mathematical expression of the above LSTM operator can be represented as:

[0035] (1)

[0036] (2)

[0037] (3)

[0038] (4)

[0039] (5)

[0040] (6)

[0041] wherein, represents the layer input data, for a batch of data, its shape is usually [batch_size (batch size), sequence_len (sequence length), feature_dim (input feature dimension)], represents the layer input data of t time step, its shape is usually [batch_size, feature_dim]; represents the hidden space state of t time step, shape is [batch_size, hidden_dim (hidden layer dimension)]; represents the weight matrix of layer input activation, the subscript "i" in the subscript represents the "input gate", "f" represents the "forget gate", "g" represents the "cell candidate value", and "o" represents the "output gate"; which specific gate (i.e., i, f, g, or o) it corresponds to, the weight matrix representing the hidden space state, bias values respectively representing the layer input activation and the hidden space mapping matrix calculation results, denotes element-wise multiplication of vectors / matrices,

[0042] Therefore, in the traditional implementation of the LSTM operator, the calculation is sequentially advanced in the time dimension in a back-and-forth dependent manner according to the above equations (1)-(6). Specifically, at each time step t, the calculation starts from the current input and the hidden space state at the previous time step to obtain the hidden space state at the time step t through equations (1)-(6).

[0043] To improve the calculation efficiency at a single time step, the implementation can use the matrix merging method shown in the following equation:

[0044] (7)

[0045] wherein the four weight matrices , , , , related to the input are vertically stacked to form a larger composite weight matrix, and the corresponding four biases , , are also stacked accordingly. Through one matrix multiplication operation and one vector addition operation, the linear intermediate results of the input gate, the forget gate, the candidate cell state, and the output gate , , , are obtained simultaneously. This optimization can reduce the overhead of multiple accesses to the weight memory and the start of the calculation operation at a single time step, to some extent, improving the hardware resource utilization of the calculation at a single time step. Subsequently, these intermediate results are added to the cyclic part calculated by and the final gating vectors , , , are generated via the activation function, and then the cell state is updated according to equations (5) and (6), and the current hidden space state is output. This will immediately serve as the necessary input for the next time step t+1 to trigger the next round of the same calculation process, and so on, until the sequence ends.

[0046] Although the matrix merging in Equation (7) optimizes the computation within a single time step, this traditional implementation is still subject to a series of fundamental problems. First, the core challenge lies in the strong temporal dependency at the algorithm level. Specifically, the computation at each time step inevitably requires the latent space state output from the previous time step as input, which constitutes an unbreakable serial chain (i.e., computation-wait-computation), making it difficult to parallelize the computation process in the time dimension, severely limiting its performance expansion on AI chips with a large number of parallel computing cores. Second, from the perspective of computational intensity and hardware utilization, although Equation (7) merges the weights, the input processed at each time step is still subject to a series of fundamental problems. Only the entire input sequence One slice (shape [batch_size, 1, feature_dim]). This makes the core operation of equation (7) essentially a matrix-vector multiplication. Even though batch processing (batch_size>1) can be regarded as a collection of multiple GEMVs, the computational cost of such operations is still relatively small compared to its memory access cost (requiring the reading of a large weight matrix), exhibiting memory bottleneck performance characteristics. It cannot generate a sufficiently dense and continuous computational flow to fully utilize the tensor cores or systolic arrays in AI accelerators that are optimized for large-scale matrix multiplication and have extremely high peak computing power, making it difficult to effectively utilize the peak computing power of the hardware. Furthermore, from the perspective of system runtime overhead analysis, since the hidden space state value update of the current time step is completed, the parallel units need to synchronize to update the state value of the next time step. In order to perform this step-by-step computation, the traditional implementation usually needs to start an independent computing kernel for each time step. Each kernel startup is accompanied by software and driver-level overhead such as task distribution and resource allocation. When the sequence is long, the cumulative overhead caused by frequent kernel startup and management will significantly affect performance improvement.

[0047] Furthermore, from the perspective of computational accuracy, AI chips, in pursuit of ultimate performance and energy efficiency, often employ lower data precision in their domain-specific architecture matrix computation units. Since the latent space state of an RNN is iteratively passed and computed between time steps, any numerical error generated at any step will be propagated and accumulated in all subsequent steps. If the chip's native low-precision computation units are used directly to implement the entire iterative process, the computational error will continuously accumulate and potentially be amplified as the sequence length increases, ultimately affecting the model's inference accuracy. This makes it difficult to guarantee the numerical reliability of the computational results while pursuing high performance.

[0048] The present disclosure provides a recurrent neural network type operator calculation method, which is based on the following. For an RNN type operator on an artificial intelligence chip, first, sequence data containing a plurality of time step input matrices is obtained, and these input matrices are subjected to single matrix-matrix multiplication operation with a layer input weight matrix, thereby generating intermediate result matrices corresponding to all time steps for gated calculation at one time. Subsequently, based on a binary semaphore mechanism, the production completion state of the batch of intermediate result matrices is synchronized among a plurality of parallel computing units of the chip. Once the production is confirmed to be completed, each computing unit performs iterative update calculation of the hidden space state in time step order using the ready intermediate result matrices, wherein in the iteration process, the production state of the latest hidden space state is also synchronized between computing units of adjacent time steps through the binary semaphore mechanism, to ensure the correctness of the time sequence dependency, and finally the hidden space state output corresponding to the entire input sequence is obtained.

[0049] Through the method of the present disclosure, the time step by step matrix-vector multiplication (GEMV) operation which is traditionally difficult to parallelize due to time sequence dependency can be reconstructed into large-scale matrix-matrix multiplication (GEMM) operation which can be executed in parallel at one time, greatly improving the calculation density and fully exerting the hardware peak computing power of the artificial intelligence chip. Secondly, by introducing the binary semaphore mechanism for synchronization within a single kernel, the data dependency between layer input mapping calculation and hidden space state iterative update calculation, and between state updates of adjacent time steps is effectively managed, which not only ensures the strict time sequence requirement of RNN iterative calculation, but also enables the state updates of different time steps to be executed in a pipelined manner on a plurality of parallel computing units, realizing parallelization in the time dimension, and completely avoiding the huge scheduling and synchronization overhead caused by multiple start-ups of independent kernels in the traditional scheme.

[0050] The recurrent neural network type operator calculation method of the present disclosure will be described in detail below with reference to Figure 2 and Figure 3 . Among them, the LSTM operator is also used as an example to introduce the method of the present disclosure, so as to be compared with the traditional implementation of the LSTM operator shown in Figure 1 . However, it should be understood that the method of the present disclosure can also be applied to the implementation of other various RNN type operators.

[0051] Figure 2 is a flowchart showing the recurrent neural network type operator calculation method 200 according to an embodiment of the present disclosure. Figure 3 is a schematic flowchart showing the implementation of the LSTM operator according to an embodiment of the present disclosure.

[0052] Through in-depth analysis of the calculation process of the RNN-type operator, the inventors of the present disclosure realized that the calculation of the gating signals (i.e. , , , ) of the core calculation unit (including the input gate, the forget gate, the candidate state, and the output gate) has a key feature: the calculation of these gating values only depends on the external input at the current time step t and the weight parameters for linear transformation of the current input, and does not directly depend on any intermediate or final calculation results generated at the previous time step t-1. Based on this, the present disclosure proposes that the overall calculation process of the RNN-type operator can be logically decoupled into two parts with different dependency attributes, and the processing of the two parts can be implemented by a fusion calculation kernel through a specific data synchronization mechanism. Specifically, as shown in Figure 3 , in the single calculation kernel of the present disclosure, the complete calculation of the RNN-type operator can be performed, wherein the complete calculation of the RNN-type operator can be divided into two stages: ① the layer input activation mapping stage for processing the linear transformation related to the current input , since the input data at each time step is independent of each other, there is no data dependency between time steps in this stage calculation, which naturally has the condition for parallelization; ② the hidden space state iterative update stage for processing the calculation related to the hidden space state at the previous time step and the update of the cell state and the hidden space state , the calculation in this stage has strict data dependency between time steps and must be performed in sequence. For Figure 2 , steps S201 to S204, wherein steps S201 to S203 can correspond to the layer input activation mapping stage, and step S204 can correspond to the hidden space state iterative update stage.

[0053] In step S201, input sequence data can be obtained, which can be an input matrix of a first number of time steps.

[0054] Optionally, input sequence data for recurrent neural network computation can be acquired (e.g., received or loaded from memory, etc.), which can contain the entire timing information to be processed, which can be composed of input matrices of the first number (i.e., sequence length) of consecutive time steps. Among different application scenarios of the RNN operator computation method of the present disclosure, the acquired input sequence data can be different types of data such as text, image, speech, video, etc., which is not limited by the present disclosure. For example, in the natural language text processing scenario, the input matrix of each time step can correspond to the feature representation of a word or sub-word after the embedding layer conversion, and the sequence constitutes a sentence or a paragraph. For another example, in the speech recognition and processing scenario, the input matrix of each time step can correspond to the acoustic features (e.g., a frame of mel-spectrogram) of a time window, and the entire sequence is a small segment of speech signal. For another example, in the video understanding and analysis scenario, the input matrix of each time step can correspond to the visual features extracted from an image frame in the video stream, and the sequence describes the visual dynamics in a period of time.

[0055] According to an embodiment of the present disclosure, acquiring the input sequence data can include: acquiring a three-dimensional input tensor with a shape of [batch size, sequence length, input feature dimension], wherein the sequence length is the first number; and for each time step in the sequence length, the input matrix of the time step is a slice of the three-dimensional input tensor at the time step, with a shape of [batch size, input feature dimension].

[0056] Optionally, the acquired input sequence data can be a three-dimensional input tensor with a shape of [batch size, sequence length, input feature dimension]. Among them, the “batch size” dimension represents the number of independent samples processed in parallel in the same calculation batch; the “sequence length” dimension corresponds to the aforementioned “first number”, which indicates the number of timing steps or elements contained in each sample; and the “input feature dimension” defines the length of the feature vector at each time step to represent the data at that moment.

[0057] Optionally, the above three-dimensional input tensor can be logically regarded as a plurality of two-dimensional matrix slices stacked along the sequence length dimension. For any time step t (t is an integer greater than or equal to 0 and less than the sequence length) in the sequence, the corresponding “input matrix” is a complete two-dimensional slice of the above three-dimensional tensor at time step t, which has a shape of [batch size, input feature dimension], wherein each row corresponds to the feature representation of a sample in the batch at time step t.

[0058] In step S202, a matrix composed of the input matrices of the first number of time steps can be subjected to a matrix multiplication operation with a layer input weight matrix to generate the first number of intermediate result matrices corresponding to the first number of time steps, which are related to the gated computation.

[0059] Optionally, for the activation mapping stage of the parallelizable layer input, the present disclosure can perform hardware-oriented optimization in depth for the linear transformation related to the current input In the traditional implementation, although the input of each time step is a tensor with shape [batch_size, 1, feature_dim] in form, since its core operation is the multiplication and addition of the weight matrix Wi and the single time step input which can be essentially regarded as a feature_dim dimensional vector, the operation at the calculation core level is essentially a GEMV operation regardless of the batch size. The calculation intensity of such GEMV operation is low, and its performance is mainly limited by the memory bandwidth, which is difficult to fully utilize the high-throughput calculation unit (such as tensor core, systolic array) designed for large-scale matrix multiplication in artificial intelligence chips. In order to overcome this performance bottleneck, the present disclosure proposes to aggregate and reorganize the multiple GEMV operations originally performed in each independent time step for a single time step input across the time dimension. Specifically, the input matrices of all time steps can be integrated in the sequence length and batch dimension to form a larger two-dimensional input matrix, and then a single large-scale GEMM operation is performed with the same weight matrix. Through this conversion, the calculation mode can be converted from multiple small-scale, memory-limited GEMV to a single calculation with extremely high density. Such large-scale GEMM operation can continuously and efficiently use the parallel calculation pipeline of the hardware, significantly improve the utilization of the calculation unit, and thus fully exert the peak computing power of the artificial intelligence chip to realize efficient inference of the RNN operator.

[0060] That is, the larger matrix formed by the integration of the input matrices of all time steps can be subjected to a one-time matrix multiplication operation with the layer input weight matrix to generate the intermediate result matrix corresponding to all time steps in parallel for the gating calculation, which can be represented as:

[0061] (8)

[0062] At this time, the layer input data can be subjected to GEMM operation with the weight matrix Wi in its entirety, thereby realizing high hardware resource utilization and exerting peak computing performance.

[0063] According to an embodiment of the present disclosure, the matrix composed of the input matrices of the first number of time steps is subjected to a matrix multiplication operation with a layer input weight matrix to generate the first number of intermediate result matrices corresponding to the first number of time steps and related to the gating calculation can comprise: flattening the three-dimensional input tensor in the batch size dimension and the sequence length dimension to obtain a two-dimensional input matrix, the shape of the two-dimensional input matrix being [batch size x sequence length, input feature dimension]; subjecting the two-dimensional input matrix to a matrix multiplication operation with the layer input weight matrix to obtain a two-dimensional result matrix, the shape of the two-dimensional result matrix being [batch size x sequence length, total gating number x hidden layer dimension]; and reorganizing the two-dimensional result matrix into a four-dimensional intermediate tensor, the shape of the four-dimensional intermediate tensor being [batch size, sequence length, total gating number, hidden layer dimension]; wherein for each time step in the sequence length, the intermediate result matrix corresponding to the time step in the first number of intermediate result matrices is a slice of the four-dimensional intermediate tensor at the time step, the shape being [batch size, total gating number, hidden layer dimension].

[0064] Optionally, the three-dimensional input tensor of the shape [batch size, sequence length, input feature dimension] obtained in step S201 can be subjected to dimension reorganization. Specifically, the batch size and the sequence length can be merged and flattened, so that the original three-dimensional tensor is converted into a two-dimensional matrix, i.e., a two-dimensional input matrix, the shape of the matrix being [batch size x sequence length, input feature dimension]. This processing can be logically equivalent to sequentially arranging the input feature vectors of all batches and all time steps into a huge two-dimensional matrix for subsequent large-scale intensive calculation.

[0065] Optionally, the two-dimensional input matrix can be subjected to large-scale dense GEMM operation with a layer input weight matrix Wi of the shape [total gating number x hidden layer dimension, input feature dimension], as shown in Figure 3 Each row in the two-dimensional result matrix can correspond to sample data of a specific batch and a specific time step in the original three-dimensional input, and after being multiplied with all weight parameters, a concatenated vector containing linear transformation results of all gating types (such as input gate, forget gate, candidate gate and output gate in LSTM) is output.

[0066] Optionally, to facilitate the subsequent state update computation iteration by time step, the two-dimensional result matrix can be reorganized back into a format with explicit batch and time structure as the layer input activation mapping result. Thus, the two-dimensional result matrix can be reshaped into a four-dimensional intermediate tensor with shape [batch size, sequence length, total gate number, hidden layer dimension], where the four-dimensional intermediate tensor clearly reconstructs the original dimensions of the data: the first dimension is the batch size, the second dimension is the sequence length (i.e., the time step), the third dimension is the gate type, and the fourth dimension is the hidden layer dimension.

[0067] Thus, for each time step t in the sequence, the intermediate result matrix (i.e., the layer input activation mapping result) corresponding thereto is the complete slice obtained when the four-dimensional intermediate tensor is indexed at t in the second dimension (sequence length). The shape of the slice is [batch size, total gate number, hidden layer dimension], which is essentially a three-dimensional tensor containing the intermediate computation results of all batch samples, all gate types, and the input-related linear transformation completed at the current time step t, which can provide independent and ready input data blocks for the subsequent iteration update depending on the hidden space state. Through the aggregation and reorganization across the time dimension as described above, the method of the present disclosure can convert a large number of dispersed, small-scale memory-constrained operations (GEMV operations) into a highly parallel, computation-intensive matrix operation, thereby fundamentally improving the utilization of hardware computing units.

[0068] Next, in step S203, the production status of the first number of intermediate result matrices can be synchronized among the multiple parallel computing units of the artificial intelligence chip based on a binary semaphore mechanism.

[0069] As described above, after the present disclosure splits the calculation of the RNN operator into two stages of layer input activation mapping and hidden space state iteration update, although the main parallelization obstacles at the algorithm level can be solved, there is a data dependency and synchronization requirement between the two stages. Specifically, the correct initiation of the hidden space state iteration update calculation depends on the completion of the layer input activation mapping calculation and ensures that all intermediate result matrices output thereby are safely and completely written into the memory or shared cache for reading by the subsequent stage. This is because the subsequent iteration update calculation needs to immediately and accurately use these intermediate results as its input. If there is a lack of synchronization mechanism, the parallel computing unit (consumer) responsible for the state update may attempt to read the data before the producer (layer input activation mapping calculation unit) has completed the work, thereby causing the reading of incorrect, partial, or inconsistent data, leading to calculation errors. Therefore, between the two decoupled calculation stages, an efficient and reliable producer-consumer synchronization mechanism must be established.

[0070] To solve this synchronization problem and avoid the overhead caused by relying on multiple kernel launches or high-cost lock mechanisms in traditional solutions, the method of the present disclosure introduces a binary semaphore mechanism. The binary semaphore mechanism is a lightweight hardware-friendly synchronization primitive, which is particularly suitable for fast state coordination among multiple parallel computing units within the same computing kernel. The semaphore is essentially a shared memory variable that can only take two states (e.g., "0" and "1"), and is combined with indivisible atomic operations to ensure the visibility and order of state changes.

[0071] According to an embodiment of the present disclosure, based on the binary semaphore mechanism, synchronizing the production state of the first number of intermediate result matrices among the multiple parallel computing units of the artificial intelligence chip can include: associating a binary semaphore with the production state of the first number of intermediate result matrices, for indicating the production state of the first number of intermediate result matrices; and setting the binary semaphore to a first value when the calculation and writing of the first number of intermediate result matrices are completed, to indicate that the production state of the first number of intermediate result matrices is in a completed state.

[0072] Optionally, a dedicated binary semaphore can be associated with the overall production state of the first number (i.e., the entire sequence) of intermediate result matrices, which serves as a global flag for uniformly indicating whether this group of data is ready. In the layer input activation mapping phase, multiple parallel computing units work together to complete the calculation and writing of all intermediate result matrices to the target memory address. Only when all these calculation and writing operations are confirmed to be completed, an atomic write operation can be performed to set the binary semaphore associated with this group of data to a preset first value (e.g., from "0" to "1"), to ensure that the declaration of the completed state is instantaneous and globally visible, thereby indicating that the production state of the first number of intermediate result matrices has entered the "completed state" as a whole.

[0073] Optionally, as another implementation manner, synchronizing the production state of the first number of intermediate result matrices based on the binary semaphore mechanism can be specifically implemented through a counting semaphore mechanism. For example, a counting semaphore with an initial value of zero can be associated with the production state of the first number of intermediate result matrices, and when each parallel computing unit participating in the calculation completes the calculation and writing of the partial intermediate result it is responsible for, an atomic increment operation can be performed on the counting semaphore. Based on this, when and only when the value of the counting semaphore reaches the preset total number of parallel computing units, it indicates that all units have completed the work, and at this time the production state can be recognized as a completed state, and the final state of the counting semaphore is equivalent to a set binary semaphore.

[0074] In step S204, in response to the production state of the first number of intermediate result matrices being a completion state, the iterative update calculation of the latent space state can be performed on the plurality of parallel computing units in time step order based on the binary semaphore mechanism using the corresponding intermediate result matrices to obtain the final latent space state output corresponding to the input sequence data, wherein the production state of the latest latent space state is synchronized between the parallel computing units for the calculation of adjacent time steps based on the binary semaphore mechanism.

[0075] Optionally, in the latent space state iterative update phase, the start condition of the latent space state iterative update calculation can be bound to the binary semaphore. According to an embodiment of the present disclosure, in response to the production state of the first number of intermediate result matrices being a completion state, the iterative update calculation of the latent space state on the plurality of parallel computing units in time step order using the corresponding intermediate result matrices can include: reading the binary semaphore by the plurality of parallel computing units, and reading the corresponding intermediate result matrices for the iterative update calculation of the latent space state when the binary semaphore is the first value.

[0076] Optionally, for each parallel computing unit used to perform the update of the latent space state, it can first read the value of the binary semaphore before starting its calculation task. The reading is an atomic read operation to ensure that the latest state is obtained. The computing units can continuously check or wait, and only when the read semaphore value is the first value, they determine that all the input data (i.e., the intermediate result matrices of the entire sequence) they need have been prepared. At this time, the computing units can safely read the corresponding intermediate result matrices and formally start the iterative update calculation of the latent space state in time step order. Through this binary semaphore mechanism, the two calculation phases before and after can be accurately and reliably coordinated with very low overhead, and the correctness of the data is guaranteed.

[0077] For the latent space state iterative update phase, due to the inherent and strict time step data dependency of its calculation process, it also needs a synchronization mechanism. Specifically, the calculation of the latent space state and the cell state of each time step t completely depends on the output and of the previous time step t-1. Under the parallel computing architecture, if multiple computing units are allocated to simultaneously calculate the states of different time steps, the unit (consumer) calculating the time step t must wait for the unit (producer) calculating the time step t-1 to complete the calculation of and Only after computation and writing can data be safely read as input. Therefore, without a synchronization mechanism, consumers within the same computation kernel may read incomplete or outdated data, leading to errors in the computation results of the entire sequence. Therefore, this disclosure applies the efficient binary semaphore mechanism, which has been verified in inter-stage synchronization, to this stage as well, to achieve lightweight, fine-grained synchronization across time steps within the same computation kernel.

[0078] According to embodiments of this disclosure, based on the binary semaphore mechanism, iteratively updating the latent space state using corresponding intermediate result matrices on multiple parallel computing units in a time-step order may include: on a first parallel computing unit used for calculation at a first time step, calculating the latest latent space state at the first time step, and synchronizing the production state of the latest latent space state at the first time step with a second parallel computing unit used for calculation at a second time step, wherein the first time step and the second time step are adjacent time steps, and the first time step precedes the second time step; on the second parallel computing unit, in response to the production state of the latest latent space state at the first time step being in a completed state, obtaining the latest latent space state at the first time step for calculating the latest latent space state at the second time step.

[0079] Optionally, such as Figure 3 As shown, data synchronization within the latent space state iteration update phase can include pipelined transmission of data ready signals indicating the completion state.

[0080] Specifically, taking the data synchronization between adjacent time steps t and t+1 (i.e., the first time step t and the second time step t+1) as an example, the first parallel computing unit (as the producer) assigned to execute the computation task of the first time step t can read the ready corresponding intermediate result matrix and, in combination with the hidden space state passed from the previous time step (the initial state if t=0), perform the complete RNN computation (e.g., the corresponding computation of formulas (1)-(6) of the LSTM operator or other operators) to calculate the latest hidden space state at the first time step t. (and cell state) After ensuring that these states are reliably written to their memory locations, the first computing unit can synchronize the latest hidden space state at the first time step with the second parallel computing unit, which is assigned to execute the computing task at the second time step t+1. The production status.

[0081] Optionally, on the second parallel computing unit (as a consumer), its computation logic and state reading directly depend on this synchronization signal. Before starting its computation task for the second time step, this unit can continuously respond (poll or wait) to the production state of the latest latent space state at the first time step. Once detecting that the production state becomes a completion state, the second parallel computing unit can determine that the required predecessor state data is already available and consistent. Based on this, it can safely acquire (read) the latest latent space state at the first time step t and After acquiring these data, the second parallel computing unit can calculate the latest latent space state at the second time step t+1 in combination with the intermediate result matrix corresponding to the second time step and . After the computation is completed, it can synchronize its production state to the unit computing the next time step (e.g., the third time step t+2) in the same way.

[0082] According to an embodiment of the present disclosure, based on the binary semaphore mechanism, the iterative update calculation of the latent space state on the plurality of parallel computing units in time step order using the corresponding intermediate result matrix can include: on a first parallel computing unit for computing a first time step, calculating the latest latent space state at the first time step using the intermediate result matrix corresponding to the first time step, and associating another binary semaphore for the latest latent space state at the first time step for indicating the production state of the latest latent space state at the first time step; after the latest latent space state at the first time step is calculated, setting the another binary semaphore to a second value to indicate that the production state of the latest latent space state at the first time step is a completion state; reading the another binary semaphore by a second parallel computing unit for computing a second time step, and when the another binary semaphore is the second value, acquiring the latest latent space state at the first time step, and calculating the latest latent space state at the second time step based on the latest latent space state at the first time step.

[0083] Optionally, the synchronization of the latent space state between the above parallel computing units can be realized by atomically modifying the binary semaphore flag associated with the latent space state, for example, setting it from the "unready state" (e.g., value 0) to the "completion state" (e.g., value 1).

[0084] Specifically, on the first parallel computing unit for the first time step (e.g. time step t) computation, after obtaining the intermediate result matrix corresponding to the first time step, the unit can associate another binary semaphore with the latest hidden space state at the first time step to be computed, for indicating the production state of the specific hidden space state, i.e. whether the data has been computed and is ready for safe reading by the subsequent time step.

[0085] Subsequently, after ensuring that these computation results are completely and correctly written into their predetermined memory locations, the first parallel computing unit can immediately perform an atomic operation, i.e. setting the another binary semaphore associated with the latest hidden space state at the first time step to a preset second value (e.g. from the initial "0" to "1"), to indicate that the production state of the latest hidden space state at the first time step has been changed to "completed state", thus making it visible to all other computing units.

[0086] The change of the production state of the latest hidden space state at the first time step can directly trigger the computation of the next time step. Specifically, the second parallel computing unit for the second time step (i.e. t+1) computation, before attempting to start the computation , must first confirm that its input data and are ready. To this end, the unit can read the another binary semaphore associated with and . Optionally, the second parallel computing unit can continuously check the value of the semaphore through polling or waiting mechanism. Only when the value of the another binary semaphore is read as the second value, the second parallel computing unit can determine that the required predecessor state is in the available completed state. At this time, it can safely obtain (i.e. read) the latest hidden space state and at the first time step, for generating the latest hidden space state and at the second time step. After the computation is completed, the second parallel computing unit can associate a new binary semaphore with and , and set it to the completed state after writing, thus passing the data ready signal to the next consumer.

[0087] Based on this, by associating independent binary semaphores with the output of each time step, an efficient computation pipeline coordinated by these lightweight flags can be constructed, so that the update computation of multiple time steps can be implemented with maximum overlap on multiple parallel computing units under the premise of ensuring strict data dependency.

[0088] Therefore, in the hidden space state iterative updating phase, by performing complete gating calculation and state transition on all time steps in the sequence in sequence, an abstract representation representing the overall information of the entire input sequence can be finally generated. When the hidden space state updating calculation of the last time step is completed, the latest hidden state obtained is the final hidden space state output corresponding to the input sequence data, which condenses all the context information and long-term dependencies of the entire sequence after network processing, filtering and integration at all time steps. It is the final and highest level of abstraction and generalization of the input sequence by the RNN operator. In tasks such as sequence classification, sentiment analysis or encoder machine translation, the final hidden space state output can usually be directly used as the feature vector of the entire sequence and input to the downstream network layer for final decision making.

[0089] According to an embodiment of the present disclosure, in the matrix operation involved in the matrix multiplication operation or the iterative updating calculation of the hidden space state, a high-precision simulation algorithm based on a low-precision hardware operation unit is used to perform calculation to meet the preset calculation precision requirement.

[0090] As described above, in order to solve the problem that the data precision (such as BF24, FP16) natively supported by the high-performance computing unit in the artificial intelligence chip may not meet the numerical stability requirement of the iterative calculation of the RNN operator, in the matrix operation involved in the above-mentioned matrix multiplication operation (such as large GEMM in step S202) or the iterative updating calculation of the hidden space state (such as the multiplication and addition operation with the hidden state weight involved in step S204), a high-precision simulation algorithm based on a low-precision hardware operation unit can be used to perform calculation, so as to utilize the hardware peak computing power while meeting the higher calculation precision (such as single-precision FP32) requirement preset at the algorithm level.

[0091] According to an embodiment of the present disclosure, performing the calculation by using the high-precision simulation algorithm based on the low-precision hardware operation unit can include: converting a first high-precision matrix involved in the matrix operation into a first low-precision matrix; calculating a first residual matrix between the first high-precision matrix and an inverse conversion result of the first low-precision matrix, and converting the first residual matrix into a second low-precision matrix; converting a second high-precision matrix involved in the matrix operation into a third low-precision matrix; calculating a second residual matrix between the second high-precision matrix and an inverse conversion result of the third low-precision matrix, and converting the second residual matrix into a fourth low-precision matrix; calculating a first product, a second product and a third product based on the low-precision hardware operation unit of the artificial intelligence chip, wherein the first product is a product of the first low-precision matrix and the third low-precision matrix, the second product is a product of the second low-precision matrix and the third low-precision matrix, and the third product is a product of the first low-precision matrix and the fourth low-precision matrix; and combining the first product, the second product and the third product to obtain a calculation result simulating a product of the first high-precision matrix and the second high-precision matrix.

[0092] Optionally, performing the calculation by using the high-precision simulation algorithm based on the low-precision hardware operation unit can first convert the high-precision input data involved in the operation in precision, for example, converting a first high-precision matrix into a first low-precision matrix which is natively supported by the hardware. Subsequently, in order to accurately compensate for the information lost due to the reduction in precision, the difference between the original high-precision matrix and the result of converting the first low-precision matrix back to the high-precision format can be calculated, thereby obtaining a residual matrix reflecting the conversion error of the first conversion, and immediately converting this residual matrix into the same low-precision format to obtain a second low-precision matrix. Similarly, a completely symmetrical operation can be performed on the second high-precision input matrix to generate a third low-precision matrix and a fourth low-precision matrix obtained from the conversion error thereof.

[0093] Based on the above four low-precision matrix data, the calculation can be performed based on the low-precision, high-performance hardware calculation unit built in the artificial intelligence chip. Specifically, the calculation unit can be sequentially called to perform three independent low-precision matrix multiplications: first, calculating the product of the first low-precision matrix and the third low-precision matrix as a base term; then, calculating the product of the second low-precision matrix and the third low-precision matrix as a first error compensation term; and finally, calculating the product of the first low-precision matrix and the fourth low-precision matrix as a second error compensation term.

[0094] Based on this, the three product results obtained from the above calculations can be summed and combined in a logic unit with higher precision accumulation capabilities. This effectively integrates the basic calculation results with two key error compensation terms, thereby generating a final result that highly simulates the multiplication operation of the original two high-precision matrices in terms of numerical accuracy. Through the above high-precision simulation algorithm, hardware units that only support low-precision native calculations can efficiently and reliably complete computational tasks with stringent precision requirements.

[0095] As a non-limiting example, operations that meet high precision (e.g., single-precision binary floating-point format FP32) can be implemented using the low-precision (e.g., dedicated, non-standard “low-precision” hardware format BF24, which is a data format designed for matrix computation units of a particular AI chip for its domain-specific architecture) and high-performance DSA matrix operation units of AI chips, as shown below:

[0096] (9)

[0097] (10)

[0098] (11)

[0099] (12)

[0100] (13)

[0101] Specifically, as shown in equations (9) and (10), the first high-precision matrix involved in the matrix operation can be... Convert to the first low-precision matrix Simultaneously calculate with by The result after inverse conversion back to high-precision format The difference between the two is used as the first residual matrix, and this residual matrix is ​​then converted into a second low-precision matrix. Symmetrically, as shown in equations (11) and (12), the second high-precision matrix can be... Convert to third low-precision matrix And calculate its corresponding residual matrix, and transform it to obtain the fourth low-precision matrix. Based on this, the low-precision hardware computing unit based on the artificial intelligence chip can sequentially calculate the product of the first low-precision matrix and the third low-precision matrix. The product of the second low-precision matrix and the third low-precision matrix and the product of the first low-precision matrix and the fourth low-precision matrix. Finally, by combining these three product results in a higher-precision accumulation unit, a calculation result that can simulate the original high-precision matrix multiplication can be obtained While fully calling the hardware peak computing power, the requirements of the algorithm on numerical precision are met.

[0102] As described above, the method of the present disclosure realizes the entire optimized calculation process in a single fusion computing kernel, which tightly integrates the originally discrete multiple calculation stages (including batch matrix multiplication (GEMM) for all time step input data, intermediate result synchronization based on binary semaphores, and hidden state iterative update in time step order) into a unified executable program unit. Based on this, the overhead of multiple kernel starts can be eliminated, and the kernel scheduling, resource allocation and context switching cost of the sequence length times in the traditional implementation mode can be reduced to once, which fundamentally solves the performance loss problem caused by frequent kernel management. Secondly, by deploying a lightweight binary semaphore mechanism inside the kernel to manage data dependencies, efficient fine-grained pipelined parallelism can be achieved, in which the calculation unit can immediately notify the next calculation unit after completing the state update of the current time step, so that the calculation of multiple time steps can be overlapped and executed on different parallel calculation units of the chip, thereby effectively breaking the serial shackles of RNN which are difficult to parallelize while ensuring the strict timing dependence of RNN. In addition, through algorithm reconstruction, the kernel aggregates a large number of scattered small-scale matrix-vector multiplications into a single large-scale matrix multiplication, so that the calculation mode is completely changed from memory bottleneck type to calculation intensive type. This can guide the hardware (especially the calculation unit of the domain-specific architecture) into its most efficient operation state, significantly improving the utilization of hardware resources and the computing power throughput. Finally, the unified execution environment provides an ideal framework for integrating high-precision simulation algorithms, so that low-precision hardware units can be seamlessly and consistently called to simulate high-precision operations during the entire calculation process, thereby ensuring the numerical stability of long sequence iterative calculation and overcoming the error accumulation problem caused by directly using low-precision hardware.

[0103] Figure 4 is a schematic diagram illustrating a recurrent neural network type operator computing apparatus 400 according to an embodiment of the present disclosure.

[0104] According to an embodiment of the present disclosure, the recurrent neural network type operator computing apparatus 400 can include a layer input obtaining module 401, a layer input mapping module 402, a layer input mapping synchronization module 403, and a hidden space state updating module 404.

[0105] The layer input obtaining module 401 can be configured to obtain input sequence data comprising a first number of time steps of input matrices. Optionally, the layer input obtaining module 401 can perform operations as described above with reference to step S201.

[0106] For example, input sequence data for recurrent neural network computation can be obtained (e.g., received or loaded from memory, etc.), which can contain the entire time series information to be processed, which can be composed of a first number (i.e., sequence length) of consecutive time steps of input matrices. Wherein, the obtained input sequence data can be a three-dimensional input tensor with shape [batch size, sequence length, input feature dimension]. Wherein, the “batch size” dimension represents the number of independent samples processed in parallel in the same computation batch; the “sequence length” dimension, i.e., corresponding to the aforementioned “first number”, indicates the number of time steps or elements contained in each sample; and the “input feature dimension” defines the length of the feature vector at each time step to represent the data at that moment.

[0107] The layer input mapping module 402 can be configured to perform matrix multiplication operation on the matrix composed of the first number of time steps of input matrices and a layer input weight matrix to generate a first number of intermediate result matrices corresponding to the first number of time steps related to gated computation. Optionally, the layer input mapping module 402 can perform operations as described above with reference to step S202.

[0108] For example, the obtained three-dimensional input tensor with shape [batch size, sequence length, input feature dimension] can be dimensionally reorganized. Specifically, the batch size and sequence length dimensions can be merged and flattened, thereby converting the original three-dimensional tensor into a two-dimensional input matrix, i.e., a two-dimensional input matrix with shape [batch size x sequence length, input feature dimension]. This processing can be logically equivalent to sequentially arranging the input feature vectors of all time steps in all batches into a huge two-dimensional matrix for subsequent large-scale intensive computation.

[0109] For example, the two-dimensional input matrix can be subjected to large-scale dense GEMM operation with a layer input weight matrix Wi with shape [total gate number x hidden layer dimension, input feature dimension], as shown in Figure 3 to obtain a two-dimensional result matrix with shape [batch size x sequence length, total gate number x hidden layer dimension]. Each row in the two-dimensional result matrix can correspond to the sample data of a specific batch and a specific time step in the original three-dimensional input, and after multiplication with all weight parameters, a concatenated vector containing the linear transformation results of all gate types (such as input gate, forget gate, candidate gate, and output gate in LSTM) is output.

[0110] For example, to facilitate the subsequent state update calculation iterating by time step, the two-dimensional result matrix can be reorganized back into a format with explicit batch and time structure as the layer input activation mapping result. Thus, the two-dimensional result matrix can be reorganized into a four-dimensional intermediate tensor with shape [batch size, sequence length, total gate number, hidden layer dimension], where the four-dimensional intermediate tensor clearly reconstructs the original dimensions of the data: the first dimension is the batch size, the second dimension is the sequence length (i.e., the time step), the third dimension is the gate type, and the fourth dimension is the hidden layer dimension.

[0111] Therefore, for each time step t in the sequence, the intermediate result matrix corresponding thereto (i.e., the layer input activation mapping result) is a complete slice of the above four-dimensional intermediate tensor indexed by t in the second dimension (sequence length). The shape of the slice is [batch size, total gate number, hidden layer dimension], which is essentially a three-dimensional tensor containing the intermediate calculation results of all batch samples, all gate types, which have completed the input-related linear transformation at the current time step t, and can provide independent and ready input data blocks for the subsequent iterative update depending on the hidden space state.

[0112] The layer input mapping synchronization module 403 can be configured to synchronize the production state of the first number of intermediate result matrices among the multiple parallel computing units of the artificial intelligence chip based on a binary semaphore mechanism. Optionally, the layer input mapping synchronization module 403 can perform the operations as described above with reference to step S203.

[0113] For example, a dedicated binary semaphore can be associated with the overall production state of the first number (i.e., the entire sequence) of intermediate result matrices, which serves as a global flag to uniformly indicate whether this set of data is ready. In the layer input activation mapping phase, multiple parallel computing units work together to complete the calculation of all intermediate result matrices and write them to the target memory address. Only when all these calculation and writing operations are confirmed to be completed, an atomic write operation can be performed to set the binary semaphore associated with this set of data to a preset first value (e.g., from "0" to "1"), to ensure that the declaration of the completion state is instantaneous and globally visible, thereby indicating that the production state of the first number of intermediate result matrices has entered the "completed state" as a whole.

[0114] The hidden space state update module 404 can be configured to, in response to the production state of the first number of intermediate result matrices being a completed state, perform, based on the binary semaphore mechanism, iterative update computation of hidden space states on the plurality of parallel computing units in time step order with the corresponding intermediate result matrices to obtain a final hidden space state output corresponding to the input sequence data, wherein the production state of the latest hidden space state is synchronized between parallel computing units for computation of adjacent time steps based on the binary semaphore mechanism. Optionally, the hidden space state update module 404 can perform operations as described above with reference to step S204.

[0115] For example, for each parallel computing unit assigned to perform the hidden space state update, it can first read the value of the binary semaphore described above before starting its computation task. The reading is an atomic read operation to ensure that the latest state is obtained. The computing unit can continuously check or wait until it determines that the read semaphore value is the first value described above, indicating that all the input data it needs (i.e. the intermediate result matrices of the entire sequence) is ready. At this time, the computing unit can safely read the corresponding intermediate result matrices and formally start the iterative update computation of hidden space states in time step order. Through this binary semaphore mechanism, the two computation phases can be accurately and reliably coordinated with very low overhead, ensuring the correctness of the data.

[0116] For example, the data synchronization in the hidden space state iterative update phase can include pipelining the data ready signal indicating the completed state.

[0117] Specifically, taking the data synchronization between adjacent time steps t and t+1 (i.e. the first time step t and the second time step t+1) as an example, the first parallel computing unit assigned to perform the computation task of the first time step t (as a producer) can read the corresponding intermediate result matrix that is ready, and combine the hidden space state passed from the previous time step (the initial state if t=0) to perform a complete RNN computation (e.g. the formulas (1)-(6) of the LSTM operator or the corresponding computation of other operators) to calculate the latest hidden space state (hid t) at the first time step t. After ensuring that these states are reliably written to their memory locations, the first computing unit can synchronize the production state of the latest hidden space state at the first time step to the second parallel computing unit assigned to perform the computation task of the second time step t+1.

[0118] Optionally, on the second parallel computing unit (as a consumer), its computation logic and state reading directly depend on this synchronization signal. This unit can continuously respond (poll or wait) to the production state of the latest latent space state at the first time step before starting its computation task for the second time step. Once it detects that the production state becomes a completion state, the second parallel computing unit can determine that its required predecessor state data is already available and consistent. Based on this, it can safely acquire (read) the latest latent space state at the first time step t and After acquiring these data, the second parallel computing unit can compute the latest latent space state at the second time step t+1 in conjunction with the intermediate result matrix corresponding to the second time step and . After the computation is completed, it can synchronize its production state to the unit computing the next time step (e.g., the third time step t+2) in the same way.

[0119] For example, the synchronization of the latent space state between the above parallel computing units can be achieved by atomically modifying the binary semaphore flag associated with the latent space state, such as setting it from an "unready state" (e.g., value 0) to a "completion state" (e.g., value 1).

[0120] Specifically, on the first parallel computing unit for the first time step (e.g., time step t) computation, after acquiring the intermediate result matrix corresponding to the first time step, the unit can associate another binary semaphore for the latest latent space state at the first time step it is going to compute, for indicating the production state of this particular latent space state, i.e., whether the data has been computed and is ready for safe reading by the subsequent time step.

[0121] Subsequently, after ensuring that these computation results are completely and correctly written into their intended memory locations, the first parallel computing unit can immediately perform an atomic operation, i.e., setting the other binary semaphore associated with the latest latent space state at the first time step to a preset second value (e.g., from the initial "0" to "1"), to indicate that the production state of the latest latent space state at the first time step has been changed to a "completion state", thus making it visible to all other computing units.

[0122] The above change of the production state of the latest latent space state at the first time step can directly trigger the computation of the next time step. Specifically, the second parallel computing unit for the second time step (i.e., t+1) computation must first confirm that its input data and and are ready before attempting to start the computation and a second binary semaphore associated with the second parallel computing unit. Optionally, the second parallel computing unit can continuously check the value of the semaphore through polling or waiting mechanism. Only when the value of the other binary semaphore is read as the second value, the second parallel computing unit can determine that the required predecessor state is in the available completion state. At this time, it can safely acquire (i.e. read) the latest latent space state at the first time step and for generating the latest latent space state at the second time step and . After the computation is completed, the second parallel computing unit can associate a new binary semaphore and and set it to the completion state after the write is completed, thus delivering the data ready signal to the next consumer.

[0123] Based on this, by associating an independent binary semaphore for the output of each time step, an efficient computation pipeline coordinated by these lightweight flags can be constructed, so that the update computation of multiple time steps can be implemented on multiple parallel computing units to the maximum extent of overlapping execution under the premise of ensuring strict data dependency.

[0124] By performing the complete gating computation and state transition in sequence for all time steps in the sequence, an abstract representation representing the overall information of the entire input sequence can be finally generated.

[0125] According to yet another aspect of the present disclosure, a recurrent neural network-like operator computing device is also provided. Figure 5 A schematic diagram of a recurrent neural network-like operator computing device 2000 according to an embodiment of the present disclosure is shown.

[0126] As shown in Figure 5 , the recurrent neural network-like operator computing device 2000 can include one or more processors 2010, and one or more memories 2020. The memory 2020 has stored therein computer-executable instructions which, when executed by the one or more processors 2010, can perform the recurrent neural network-like operator computing method as described above.

[0127] The processor in the embodiments of the present disclosure can be an integrated circuit chip with a processing capability of signals. The processor can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can be any conventional processor or the like, which can be X86 architecture or ARM architecture.

[0128] Generally, various example embodiments of the present disclosure can be implemented in hardware or special-purpose circuitry, software, firmware, logic, or any combination thereof. Certain aspects can be implemented in hardware, while other aspects can be implemented in firmware or software which can be executed by a controller, microprocessor or other computing device, X86 architecture or ARM architecture.

[0129] For example, the method or device according to the embodiments of the present disclosure can also be implemented by means of the architecture of the computing device. The computing device can include a bus, one or more CPUs, read-only memory (ROM), random access memory (RAM), a communication port connected to a network, an input / output component, a hard disk, etc. The storage device in the computing device, such as ROM or hard disk, can store various data or files used for processing and / or communication of the recurrent neural network class operator calculation method provided by the present disclosure and the program instructions executed by the CPU. The computing device can also include a user interface. Of course, the above architecture is only exemplary, and when implementing different devices, one or more components in the above computing device can be omitted according to actual needs.

[0130] According to another aspect of the present disclosure, a computer readable storage medium is also provided. The computer readable storage medium has computer executable instructions stored thereon. When the computer executable instructions are run by a processor, the computer executable instructions can perform the recurrent neural network type operator computing method according to the embodiments of the present disclosure described with reference to the above figures. The computer readable storage medium in the embodiments of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct Rambus dynamic random access memory (DRAM). It should be noted that the memory of the methods described herein is intended to include, but not be limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0131] The embodiments of the present disclosure also provide a computer program product or a computer program, which includes computer executable instructions stored in a computer readable storage medium. A processor of a computer device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the computer device performs the recurrent neural network type operator computing method according to the embodiments of the present disclosure.

[0132] The embodiments of the present disclosure provide a recurrent neural network type operator computing method, apparatus, device, computer program product, and computer readable storage medium.

[0133] The method provided by the embodiments of the present disclosure is directed to an RNN type operator on an artificial intelligence chip. First, sequence data containing a plurality of time step input matrices is acquired, and the input matrices as a whole are subjected to single matrix-matrix multiplication operation with a layer input weight matrix, thereby generating, at one time, intermediate result matrices corresponding to all time steps for gated calculation. Subsequently, based on a binary semaphore mechanism, the production completion state of the batch of intermediate result matrices is synchronized among a plurality of parallel computing units of the chip. Once the production is confirmed to be completed, each computing unit performs iterative update calculation of the hidden space state in time step order using the ready intermediate result matrices, wherein, in the iteration process, the production state of the latest hidden space state is also synchronized between computing units of adjacent time steps through the binary semaphore mechanism, to ensure the correctness of the time sequence dependency, and finally the hidden space state output corresponding to the entire input sequence is obtained. Through the method of the embodiments of the present disclosure, the matrix-vector multiplication (GEMV) operation which is traditionally difficult to parallelize due to time sequence dependency and is performed in time step series, is restructured into large-scale matrix-matrix multiplication (GEMM) operation which can be executed in parallel at one time, greatly improving the calculation density and thus fully exerting the hardware peak computing power of the artificial intelligence chip. Secondly, by introducing the binary semaphore mechanism for synchronization within a single kernel, the data dependency between the layer input mapping calculation and the iterative update calculation of the hidden space state, and between the state update of adjacent time steps, is effectively managed, which not only ensures the strict time sequence requirement of RNN iterative calculation, but also enables the state update of different time steps to be executed in a pipelined manner on a plurality of parallel computing units, realizing parallelization in the time dimension, and completely avoiding the huge scheduling and synchronization overhead caused by multiple start of independent kernels in the traditional scheme.

[0134] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectural, functional, and operational architectures of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a portion of code that contains at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different orders than that shown in the accompanying drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, or they can be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer executable instructions.

[0135] In general, the various example embodiments of the disclosure can be implemented in hardware or special-purpose circuits, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software which can be executed by a controller, microprocessor or other computing device, Although the various example embodiments of the disclosure can be illustrated and described as block diagrams, flow charts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques or methods described herein can be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controler or other computing devices, or some combination thereof.

[0136] The example embodiments of the disclosure described in detail above are merely illustrative, and not restrictive. Those skilled in the art should understand that various modifications and combinations can be made to these embodiments or features thereof without departing from the principles and spirit of the disclosure, and such modifications should fall within the scope of the disclosure.

Claims

1. A method for calculating operators in recurrent neural networks, characterized in that, The method includes: Obtain input sequence data, wherein the input sequence data includes an input matrix for a first number of time steps; The matrix composed of the input matrices of the first number of time steps is multiplied with the layer input weight matrix to generate the first number of intermediate result matrices related to the gating computation, which correspond to the first number of time steps. Based on the binary semaphore mechanism, the production status of the first number of intermediate result matrices is synchronized among multiple parallel computing units of the artificial intelligence chip. In response to the completion of the production state of the first number of intermediate result matrices, based on the binary semaphore mechanism, the latent space state is iteratively updated on the multiple parallel computing units according to the time step order using the corresponding intermediate result matrices to obtain the final latent space state output corresponding to the input sequence data. Among the parallel computing units used for calculations in adjacent time steps, the production state of the latest latent space state is synchronized based on the binary semaphore mechanism.

2. The method as described in claim 1, characterized in that, Based on the binary semaphore mechanism, synchronizing the production status of the first number of intermediate result matrices among multiple parallel computing units of an artificial intelligence chip includes: A binary semaphore is associated with the production status of the first number of intermediate result matrices to indicate the production status of the first number of intermediate result matrices; Once the calculation and writing of the first number of intermediate result matrices are completed, the binary semaphore is set to a first value to indicate that the production status of the first number of intermediate result matrices is complete. Wherein, in response to the production state of the first number of intermediate result matrices being in a completed state, the iterative update calculation of the latent space state using the corresponding intermediate result matrices on the multiple parallel computing units in a time-step sequence includes: The binary semaphore is read by the multiple parallel computing units, and when the binary semaphore is the first value, the corresponding intermediate result matrix is ​​read for iterative update calculation of the latent space state.

3. The method as described in claim 1, characterized in that, Based on the aforementioned binary semaphore mechanism, the iterative update calculation of the latent space state using the corresponding intermediate result matrix on multiple parallel computing units in sequence according to the time step includes: On a first parallel computing unit used for computation at a first time step, the latest hidden space state at the first time step is computed, and the production state of the latest hidden space state at the first time step is synchronized to a second parallel computing unit used for computation at a second time step, wherein the first time step and the second time step are adjacent time steps, and the first time step precedes the second time step. In the second parallel computing unit, in response to the production state of the latest hidden space state at the first time step being completed, the latest hidden space state at the first time step is obtained and used to calculate the latest hidden space state at the second time step.

4. The method as described in claim 1, characterized in that, Based on the aforementioned binary semaphore mechanism, the iterative update calculation of the latent space state using the corresponding intermediate result matrix on multiple parallel computing units in sequence according to the time step includes: On the first parallel computing unit used for the calculation of the first time step, the latest hidden space state at the first time step is calculated using the intermediate result matrix corresponding to the first time step, and another binary semaphore is associated with the latest hidden space state at the first time step to indicate the production state of the latest hidden space state at the first time step. After calculating the latest hidden space state at the first time step, the other binary semaphore is set to a second value to indicate that the production state of the latest hidden space state at the first time step is a completed state. The second parallel computing unit for the calculation of the second time step reads the other binary semaphore, and when the other binary semaphore is the second value, obtains the latest hidden space state at the first time step, and calculates the latest hidden space state at the second time step based on the latest hidden space state at the first time step.

5. The method as described in claim 1, characterized in that, In the matrix operations involved in the matrix multiplication operation or the iterative update calculation of the latent space state, a high-precision simulation algorithm based on a low-precision hardware computing unit is used to perform the calculation to meet the preset calculation accuracy requirements.

6. The method as described in claim 5, characterized in that, The calculations performed using a high-precision simulation algorithm based on a low-precision hardware processing unit include: Convert the first high-precision matrix involved in matrix operations into the first low-precision matrix; Calculate the first residual matrix between the inverse transformation results of the first high-precision matrix and the first low-precision matrix, and convert the first residual matrix into a second low-precision matrix; Convert the second high-precision matrix involved in matrix operations into a third low-precision matrix; Calculate the second residual matrix between the inverse transformation result of the second high-precision matrix and the third low-precision matrix, and convert the second residual matrix into a fourth low-precision matrix; Based on the low-precision hardware computing unit of the artificial intelligence chip, a first product, a second product, and a third product are calculated, wherein the first product is the product of the first low-precision matrix and the third low-precision matrix, the second product is the product of the second low-precision matrix and the third low-precision matrix, and the third product is the product of the first low-precision matrix and the fourth low-precision matrix. The first product, the second product, and the third product are combined to obtain the calculation result of simulating the product of the first high-precision matrix and the second high-precision matrix.

7. The method as described in claim 1, characterized in that, Obtaining input sequence data includes: Obtain a three-dimensional input tensor with shape [batch size, sequence length, input feature dimension], wherein the sequence length is the first quantity; Wherein, for each time step in the sequence length, the input matrix of the time step is a slice of the three-dimensional input tensor at that time step, and its shape is [batch size, input feature dimension].

8. The method as described in claim 7, characterized in that, Perform matrix multiplication on the matrix composed of the input matrices of the first number of time steps and the layer input weight matrix to generate the first number of intermediate result matrices related to the gating computation, corresponding to the first number of time steps, including: The three-dimensional input tensor is flattened in the batch size dimension and the sequence length dimension to obtain a two-dimensional input matrix. The shape of the two-dimensional input matrix is ​​[batch size × sequence length, input feature dimension]; Perform matrix multiplication on the two-dimensional input matrix and the layer input weight matrix to obtain a two-dimensional result matrix. The shape of the two-dimensional result matrix is ​​[batch size × sequence length, total number of gates × hidden layer dimension]. The two-dimensional result matrix is ​​reorganized into a four-dimensional intermediate tensor, the shape of which is [batch size, sequence length, total number of gates, hidden layer dimension]; Wherein, for each time step in the sequence length, the intermediate result matrix corresponding to the time step in the first number of intermediate result matrices is a slice of the four-dimensional intermediate tensor at the time step, and its shape is [batch size, total number of gates, hidden layer dimension].

9. A computational device for recurrent neural network operators, characterized in that, The device includes: The layer input acquisition module is configured to acquire input sequence data, the input sequence data including an input matrix of a first number of time steps; The layer input mapping module is configured to perform matrix multiplication on a matrix composed of the input matrices of the first number of time steps and the layer input weight matrix to generate the first number of intermediate result matrices related to the gating computation, corresponding to the first number of time steps. The layer input mapping synchronization module is configured to synchronize the production state of the first number of intermediate result matrices among multiple parallel computing units of the artificial intelligence chip based on a binary semaphore mechanism; and The latent space state update module is configured to respond to the production state of the first number of intermediate result matrices being in a completed state. Based on the binary semaphore mechanism, iterative update calculations of the latent space state are performed on the multiple parallel computing units in time step order using the corresponding intermediate result matrices to obtain the final latent space state output corresponding to the input sequence data. Among the parallel computing units used for calculations in adjacent time steps, the production state of the latest latent space state is synchronized based on the binary semaphore mechanism.

10. A computational device for recurrent neural network operators, characterized in that, The device includes: One or more processors; and One or more memories storing computer-executable instructions, which, when executed by the processor, perform the method of any one of claims 1-8.

11. A computer program product, characterized in that, The computer program product is stored on a computer-readable storage medium and includes computer-executable instructions that, when executed by a processor, cause a data processing device to perform the method of any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when executed by a processor, are used to implement the method of any one of claims 1-8.

Images