Multi-input-output transformer model chip architecture and computing method

Abstract

The present application belongs to the field of integrated circuit chip design and artificial intelligence technology, and provides a multi-input and output Transformer model chip architecture and a calculation method to solve the problem that current chips are difficult to perform professional acceleration calculation on large model operators. The multi-input and output Transformer model chip architecture comprises an encoder module, a decoder module, a linear calculation module, a softmax calculation module and a pipeline stage controller. The pipeline stage controller is configured to control the synchronization of the input data stream of the encoder module and the input data stream of the decoder module, and the synchronization of the same level data stream of the multi-head attention acceleration calculation module in the encoder module and the multi-head attention acceleration calculation module in the decoder module, so as to realize the pipeline stage calculation of the entire Transformer model, thereby improving the calculation efficiency of the Transformer model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of integrated circuit chip design and artificial intelligence, and particularly relates to a multi-input and output Transformer model chip architecture and a calculation method. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] The Transformer model is the core basis of large model calculation. The attention mechanism is an important component of the Transformer model, which is a basic calculation model of a large model. It can effectively capture the dependency between contexts, adaptively integrate local features and global dependencies; in the target detection field, the attention graph is inferred along the channel and spatial paths, and the hierarchical attention is constructed in the model bottleneck; in the image segmentation field, it can capture the long dependency of all information on the current information, better understand and integrate information of different modalities, thereby improving the ability to handle complex scenarios; in the image recognition field, the position information is integrated into the channel attention to generate a direction-aware and position-sensitive attention graph, enhancing the performance of mobile networks in image classification and downstream tasks; in the speech segmentation field, an accumulated attention graph is maintained for each target class of the training data, and the training attention of different stages is accumulated to obtain more complete information graphs.

[0004] Most of the existing large model acceleration calculation chips are based on general-purpose graphics processing chips or general-purpose AI SoC (Artificial Intelligence System on Chip) chips for calculation, and rely on the software compilation system in the corresponding chips, lack of specialized hardware circuit calculation operators such as normalization and specialized software programming system optimization, and are difficult to perform professional acceleration calculation for large model operators. SUMMARY

[0005] In order to solve the technical problems existing in the background art, the present application provides a multi-input and output Transformer model chip architecture and a calculation method, which can improve the calculation efficiency of the Transformer model.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] The first aspect of the present application provides a multi-input and output Transformer model chip architecture.

[0008] A multi-input-output Transformer model chip architecture, comprising: an encoder module, a decoder module, a linear calculation module, a softmax calculation module, and a pipeline stage controller;

[0009] The encoder module is configured to encode an input signal and transmit the encoded signal to the decoder module; the decoder module is configured to generate a decoded signal according to the output signal of the encoder module and a target signal; the linear calculation module is configured to perform linear mapping calculation on the decoded signal, and the softmax calculation module outputs the decoded signal with the maximum prediction probability;

[0010] The encoder module and the decoder module each include a multi-head attention acceleration calculation module.

[0011] The pipeline stage controller is configured to synchronize the input data stream of the encoder module with the input data stream of the decoder module, and synchronize the same level data stream of the multi-head attention acceleration calculation module in the encoder module with the multi-head attention acceleration calculation module in the decoder module, so as to realize the pipeline stage calculation of the entire Transformer model.

[0012] As an implementation, the multi-head attention acceleration calculation module is configured to calculate the attention weight of each input signal position with respect to all the remaining signal positions; each multi-head attention acceleration calculation module independently calculates the input signal sequence with different parameters, and simultaneously calculates the output of different feature values for the input signal sequence.

[0013] As an implementation, the multi-head attention acceleration calculation module is a calculation array architecture composed of m*n attention calculation modules; m and n are both positive integers, and at least one of them is greater than or equal to 2.

[0014] As an implementation, the multi-head attention acceleration calculation module calculates and outputs the multi-head feature value of one input signal in each clock cycle.

[0015] As an implementation, the encoder module includes an encoding input data module, a multi-head attention acceleration calculation module, an encoding feedforward calculation module, and an encoding normalization module; the output ends of the multi-head attention acceleration calculation module and the encoding feedforward calculation module of the encoder module are connected to an encoding normalization module.

[0016] The decoder module comprises a decoding input data module, two multi-head attention acceleration calculation modules, a decoding normalization module and a decoding feedforward calculation module; the two multi-head attention acceleration calculation modules of the decoder module are connected in series, and the output end of each multi-head attention acceleration calculation module is connected with a decoding normalization module; wherein the input end of the first multi-head attention acceleration calculation module of the decoder module is connected with the decoding input data module, and the decoding normalization module connected with the second multi-head attention acceleration calculation module of the decoder module is connected with the decoding feedforward calculation module.

[0017] As an implementation form, the encoding normalization module and the encoding feedforward calculation module of the encoder module constitute the same level data stream.

[0018] As an implementation form, the encoding normalization module connected with the output end of the encoding feedforward calculation module and the second multi-head attention acceleration calculation module of the decoder module adopt a pipeline stage synchronization mode to complete data calculation.

[0019] As an implementation form, the decoding normalization module and the decoding feedforward calculation module connected with the second multi-head attention acceleration calculation module of the decoder module constitute the same level data stream; the decoding normalization module and the linear calculation module of the decoder module constitute the same level data stream.

[0020] As an implementation form, the last stage data stream controlled by the pipeline stage controller is a softmax calculation data stream, and the final output is the calculation data of the Transformer.

[0021] The second aspect of the present application provides a calculation method of a multi-input and output Transformer model chip architecture.

[0022] A calculation method of a multi-input and output Transformer model chip architecture, comprising:

[0023] The input data stream of the encoder module and the input data stream of the decoder module are controlled to be synchronized by using a pipeline stage controller;

[0024] The input signal is encoded by using the encoder module and transmitted to the decoder module;

[0025] The decoding signal is generated by the decoder module according to the output signal of the encoder module and the target signal;

[0026] The decoding signal is linearly mapped and calculated by using the linear calculation module, and the decoding signal with the maximum prediction probability is output by the softmax calculation module;

[0027] In the process of encoding the input signal by the encoder module and decoding by the decoder module, the same level data stream synchronization of the multi-head attention acceleration calculation module in the encoder module and the multi-head attention acceleration calculation module in the decoder module is controlled by using the pipeline stage controller.

[0028] The beneficial effects of the present application are:

[0029] The pipeline stage controller of the present application controls the input data stream synchronization of the encoder module and the input data stream synchronization of the decoder module, and the same level data stream synchronization of the multi-head attention acceleration calculation module in the encoder module and the multi-head attention acceleration calculation module in the decoder module, which meets the requirements of circuit timing calculation and Transformer model calculation by using pipeline stage synchronization control, and realizes the pipeline stage calculation of the entire Transformer model.

[0030] The present application optimizes the composition of the encoder module, directly normalizes the multi-head attention acceleration calculation result of the encoder module and inputs it to the second multi-head attention acceleration calculation module of the decoder, thereby reducing one level of bubble pipeline stage and improving the calculation efficiency of the Transformer model.

[0031] The present application simplifies the design of the feedforward calculation module of the encoder, and synchronizes it with the multi-head attention acceleration calculation module of the decoder in terms of circuit timing requirements, which not only meets the performance requirements of reducing bubble pipeline stage and the requirements of circuit timing calculation, but also meets the requirements of Transformer model calculation.

[0032] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0033] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation of the present application.

[0034] Figure 1 is a multi-input-output Transformer model chip architecture schematic diagram of an embodiment of the present application;

[0035] Figure 2 is a multi-head attention acceleration calculation module architecture diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described below in combination with the drawings and embodiments.

[0037] It should be noted that the following detailed description is illustrative only, and is intended to provide further description in order to provide a thorough understanding of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains.

[0038] It is also important to note that the use of the terms herein are intended to be illustrative only and are not intended to limit the exemplary embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0039] According to Figure 1 As shown in the figure, a multi-input-output Transformer model chip architecture of an embodiment of the present application comprises an encoder module, a decoder module, a linear calculation module, a softmax calculation module, and a pipeline stage controller.

[0040] The encoder module is configured to encode an input signal and transmit the encoded signal to the decoder module. The decoder module is configured to generate a decoded signal based on an output signal of the encoder module and a target signal. The linear calculation module is configured to perform linear mapping calculation on the decoded signal, and output a decoded signal with the maximum prediction probability via the softmax calculation module. The encoder module and the decoder module each comprise a multi-head attention acceleration calculation module. The pipeline stage controller controls the pipeline stage calculation of the entire Transformer model. Specifically, the pipeline stage of input data of the encoder module is synchronized with the pipeline stage of input data of the decoder module, and the pipeline stage of the multi-head attention acceleration calculation module in the encoder module is synchronized with the pipeline stage of the multi-head attention acceleration calculation module in the decoder module.

[0041] Specifically, the encoder module comprises an encoded input data module, a multi-head attention acceleration calculation module, an encoded feedforward calculation module, and an encoded normalization module. The output ends of the multi-head attention acceleration calculation module and the encoded feedforward calculation module of the encoder module are connected to an encoded normalization module.

[0042] The decoder module comprises a decoded input data module, two multi-head attention acceleration calculation modules, a decoded normalization module, and a decoded feedforward calculation module. The two multi-head attention acceleration calculation modules of the decoder module are connected in series, and the output ends of each multi-head attention acceleration calculation module are connected to a decoded normalization module. The input end of the first multi-head attention acceleration calculation module of the decoder module is connected to the decoded input data module, and the decoded normalization module connected to the second multi-head attention acceleration calculation module of the decoder module is connected to the decoded feedforward calculation module.

[0043] In the embodiment of the present application, each multi-head attention acceleration calculation module independently calculates the input signal sequence with different parameters, and simultaneously calculates different characteristic values of the input signal sequence, thereby increasing the expression dimension of the model for signal characteristics. The calculation process of the multi-head attention acceleration calculation module is as follows:

[0044] (1);

[0045] (2);

[0046] (3);

[0047] wherein Q, K, and V represent query vector (Query), key vector (Key), and value vector (Value), respectively;

[0048] Concat represents a concatenation operation, represents the calculation result of the i-th attention, h is an output matrix weight matrix; represents the dimension of the key vector K, which is used to scale the attention weight and change the attention matrix into a standard normal distribution; represents a multi-head attention acceleration calculation function; represents an attention function; , , and represent the weights of the query vector, the key vector, and the value vector corresponding to the i-th attention, respectively; T represents the transpose of a vector; h represents the transpose of the key vector K.

[0049] In the embodiment of the present application, as shown in Figure 2 , the multi-head attention acceleration calculation module is a calculation array architecture composed of m*n attention calculation modules; m and n are both positive integers, and any one of them is at least greater than or equal to 2. Figure 2 Head mn in the formula represents the attention head in the m-th row and the n-th column, Figure 2 ​The W SRAM in the formula (3) represents a parameter storage unit. The multi-head attention acceleration calculation module is composed of an input data module, a data flow controller, a pipeline control bus module, a parameter transmission distribution bus module, an output data bus module and a calculation array unit. The calculation array unit of the embodiment of the present application is composed of an attention calculation unit and its corresponding parameter storage unit. Each attention calculation unit performs the calculation of the formula (3), and reads parameter data from the parameter storage unit corresponding to each attention calculation unit. The parameter storage unit is an on-chip SRAM (Static Random-Access Memory), and the parameter data is written into the corresponding parameter storage unit according to the agreed data format through the parameter transmission distribution bus module during the initial loading process of the circuit, which is controlled by the input data module and the data flow controller. The pipeline control bus module is used to control the pipeline level calculation of the data in the multi-head attention acceleration calculation module.

[0050] The multi-head attention acceleration calculation module calculates and outputs one multi-head feature value of the input signal to the output data bus module in each clock cycle. The output data bus module is connected with a corresponding normalization module.

[0051] The data flow controller of the embodiment of the present application controls the writing of the parameter data of the multi-head attention acceleration calculation module, the reading of the input signal from the input data module, and the task of delivering data to each attention calculation unit. The data flow controller can be realized by a special ASIC circuit, or can be realized by an MCU controller program. The MCU includes but is not limited to an ARM, a RISC-V instruction set architecture microcontroller.

[0052] In the embodiment of the present application, the encoding feedforward calculation module and the decoding feedforward calculation module are each composed of two fully connected layers, which can be independently applied to each position of the signal, and are connected through a ReLU activation function. The encoding feedforward calculation module and the decoding feedforward calculation module are specifically expressed as follows:

[0053] (4);

[0054] wherein, is the input data; is a ReLU activation function; , is a weight matrix of the two fully connected layers, , is a bias vector.

[0055] The pipeline controller of the application controls the pipeline calculation of the whole Transformer model calculation. The input data stream of the encoder module is synchronized with the input data stream of the decoder module; the multi-head attention acceleration calculation module in the encoder module is synchronized with the same level data stream of the multi-head attention acceleration calculation module in the decoder module; the normalization and feedforward calculation module of the encoder module constitutes the same level data stream, and there is no corresponding parallel pipeline calculation in the decoder module, at this time, a bubble is generated in the decoder module; the output of the feedforward calculation module of the encoder module is input to the normalization module of the decoder module, and then input to the second multi-head attention acceleration calculation module of the decoder module, and the normalization module and the second multi-head attention acceleration calculation module of the decoder module adopt a pipeline synchronization method to complete data calculation; the normalization and feedforward calculation module after the second multi-head attention acceleration calculation module of the decoder module constitutes the same level data stream; the normalization module and the linear calculation module of the subsequent decoder module constitute the same level data stream; the last level data stream controlled by the pipeline controller is the softmax calculation data stream, and the final output is the calculation data of the Transformer.

[0056] There is a dependency between the second multi-head attention acceleration calculation module of the decoder module and the first multi-head attention acceleration calculation module of the encoder, which causes a bubble in the pipeline calculation. Moreover, the feedforward calculation pipeline after the second multi-head attention acceleration calculation module of the decoder module is also a bubble pipeline. Therefore, the encoder module is optimized, and the multi-head attention acceleration calculation result of the encoder module is normalized and then input to the second multi-head attention acceleration calculation module of the decoder, thereby reducing a level of bubble pipeline and improving the calculation efficiency of the Transformer model. The feedforward calculation module of the encoder is simplified accordingly, and is synchronized with the multi-head attention acceleration calculation module of the decoder in circuit timing requirements, which meets the performance requirements of reducing the bubble pipeline, the requirements of circuit timing calculation, and the requirements of Transformer model calculation.

[0057] In one or more embodiments, a calculation method of a multi-input-output Transformer model chip architecture includes:

[0058] The input data stream of the encoder module is synchronized with the input data stream of the decoder module by using the pipeline controller;

[0059] The input signal is encoded by using the encoder module and transmitted to the decoder module;

[0060] The decoder module generates a decoding signal according to the output signal of the encoder module and a target signal;

[0061] The linear computation module is used for linear mapping computation on the decoded signal, and the decoded signal with the maximum prediction probability is output through the softmax computation module;

[0062] In the process of encoding the input signal by the encoder module and decoding by the decoder module, the same level data flow synchronization of the multi-head attention acceleration computation module in the encoder module and the multi-head attention acceleration computation module in the decoder module is controlled by using the pipeline stage controller.

[0063] The above only is the preferred embodiment of the present application, and is not used for limiting the present application. For the person skilled in the art, the present application can have various changes and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A multi-input / output Transformer model chip, characterized in that, include: Encoder module, decoder module, linear calculation module, softmax calculation module, and pipeline controller; The encoder module is used to encode the input signal and transmit it to the decoder module; The decoder module is used to generate a decoded signal based on the output signal of the encoder module and the target signal; the linear calculation module is used to perform linear mapping calculation on the decoded signal, and the softmax calculation module outputs the decoded signal with the highest prediction probability. Both the encoder module and the decoder module include a multi-head attention acceleration computing module; The pipeline controller is configured to: synchronize the input data stream of the encoder module with the input data stream of the decoder module, and synchronize the data stream of the multi-head attention acceleration computation module in the encoder module with the multi-head attention acceleration computation module in the decoder module at the same level, so as to realize pipeline computation of the entire Transformer model. The encoder module includes an encoding input data module, a multi-head attention acceleration calculation module, an encoding feedforward calculation module, and an encoding normalization module; the outputs of the multi-head attention acceleration calculation module and the encoding feedforward calculation module of the encoder module are both connected to an encoding normalization module. The decoder module includes a decoding input data module, two multi-head attention acceleration computation modules, a decoding normalization module, and a decoding feedforward computation module; The two multi-head attention acceleration computation modules of the decoder module are connected in series, and the output of each multi-head attention acceleration computation module is connected to a decoding normalization module; wherein, the input of the first multi-head attention acceleration computation module of the decoder module is connected to the decoding input data module, and the decoding normalization module connected to the second multi-head attention acceleration computation module of the decoder module is connected to the decoding feedforward computation module. The decoding normalization module and the decoding feedforward calculation module connected to the second multi-head attention acceleration calculation module of the decoder module form a data flow at the same level; the decoding normalization module and the linear calculation module of the decoder module form a data flow at the same level.

2. The multi-input / output Transformer model chip as described in claim 1, characterized in that, The multi-head attention acceleration calculation module is used to calculate the attention weight of the position of each input signal and the positions of all other signals; each multi-head attention acceleration calculation module independently calculates different parameters for the input signal sequence, and simultaneously calculates and outputs different feature values ​​for the input signal sequence.

3. The multi-input / output Transformer model chip as described in claim 1, characterized in that, The multi-head attention acceleration computing module is a computing array architecture consisting of m*n attention computing modules; m and n are both positive integers, and either one is at least greater than or equal to 2.

4. The multi-input / output Transformer model chip as described in claim 1, characterized in that, The multi-head attention acceleration computing module calculates and outputs a multi-head feature value of an input signal in each clock cycle.

5. The multi-input / output Transformer model chip as described in claim 1, characterized in that, The encoder module's encoding normalization module and encoding feedforward calculation module form a data stream at the same level.

6. The multi-input / output Transformer model chip as described in claim 1, characterized in that, The output of the encoding feedforward computation module is connected to the encoding normalization module and the second multi-head attention acceleration computation module of the decoder module, which use a pipelined synchronous method to complete the data computation.

7. The multi-input / output Transformer model chip as described in claim 1, characterized in that, The last stage of the pipeline controller controls the softmax computation data stream, which ultimately outputs the Transformer's computation data.

8. A computational method based on the multi-input / output Transformer model chip architecture as described in any one of claims 1-7, characterized in that, include: A pipelined controller is used to synchronize the input data stream of the encoder module with the input data stream of the decoder module. The input signal is encoded using the encoder module and then transmitted to the decoder module. The decoder module generates a decoded signal based on the output signal of the encoder module and the target signal. The linear calculation module performs linear mapping calculations on the decoded signal, and the softmax calculation module outputs the decoded signal with the highest prediction probability. In the process of encoding the input signal by the encoder module and decoding the input signal by the decoder module, a pipelined controller is used to control the synchronization of the data flow at the same level between the multi-head attention acceleration calculation module in the encoder module and the multi-head attention acceleration calculation module in the decoder module.

Images