Deep learning processor architecture system and application methods with attention mechanism

Abstract

This application discloses a deep learning processor architecture system and application method incorporating an attention mechanism, primarily relating to the field of processor architecture technology. It addresses the problems of excessive load caused by traditional architectures where execution is handled by the host processor, separation of the attention mechanism from the processor's computational unit, and complex architecture design. The system includes: a real-time calculation unit for attention mechanism parameters comprising: an input sequence module; a decomposition method reading module connected to the input sequence module; an attention parameter generation module and an input sequence buffer module connected to the decomposition method reading module; an attention score calculation and control FSM state machine module connected to the attention parameter generation module; an attention score calculation module connected to the attention score calculation and control FSM state machine module; a MUX data merging module connected to the input sequence buffer module and the attention score calculation and control FSM state machine module; and an optimal scheduling method selection and setting module connected to the MUX data merging module.

Description

Technical Field

[0001] This application relates to the field of deep learning processor architecture technology, and in particular to a deep learning processor architecture system with attention mechanism and its application method. Background Technology

[0002] A processor, a collective term for either a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), is a core component of modern computer systems. Its main functions are parsing computer instructions and rapidly processing complex data. With the rapid advancement of current technology, the demand for computational performance in deep learning processors is increasing, leading researchers to propose incorporating attention mechanisms into deep learning.

[0003] However, this design also has some problems in practical applications, specifically: 1. Traditional architectures require the host processor to perform the splitting of the data to be processed and the calculation of attention scores, and to perform specific scheduling of the target computing units for task allocation. These processes bring extremely high loads to the processor, and excessive loads can lead to excessively high computational latency in other important processes of the processor; 2. Existing solutions usually adopt an organizational architecture that separates the attention mechanism from the processor computing units. That is, the user's instructions and the data to be processed need to first go through the host processor to perform attention score calculations, and the results are then transmitted to the processor computing unit module for target data processing via the on-chip bus after scheduling. This design will inevitably reduce the data processing efficiency of important processes; 3. Existing architectures are complex and extremely difficult to apply. Users need to master complex programming and development basic systems, which is not conducive to the application and development of deep learning systems by a wide range of researchers. Summary of the Invention

[0004] This application provides a deep learning processor architecture system and application method with an attention mechanism to solve the problems of excessive load caused by the host processor in traditional architecture, separation of the attention mechanism from the processor computing unit, and complex architecture design.

[0005] Firstly, this application provides a deep learning processor architecture system with an attention mechanism, the system comprising:

[0006] The deep learning processor includes: an attention mechanism control module, a computing unit control module, a processor configuration module, and an AXI (Advanced eXtensible Interface) control / data bus connecting the various modules; and the attention mechanism control module and the computing unit control module adopt an integrated design.

[0007] The attention mechanism control module includes: a real-time calculation unit for attention mechanism parameters, an implicit attention mechanism scheduling unit, and an explicit attention mechanism scheduling unit.

[0008] The real-time attention mechanism parameter calculation unit includes: an input sequence module, a decomposition method reading module connected to the input sequence module, an attention parameter generation module and an input sequence buffer module connected to the decomposition method reading module, an attention score calculation and control FSM (Finite State Machine) state machine module connected to the attention parameter generation module, an attention score calculation module connected to the attention score calculation and control FSM state machine module, an MUX (Multiplexer) data merging module connected to the input sequence buffer module and the attention score calculation and control FSM state machine module, an optimal scheduling method selection and setting module connected to the MUX data merging module, an attention score output module connected to the optimal scheduling method selection and setting module, and a management information configuration module connected to all modules; the management information configuration module is connected to the processor configuration module, and the attention score output module is connected to the implicit attention mechanism scheduling unit and the explicit attention mechanism scheduling unit.

[0009] In one implementation of this application, the system further includes:

[0010] The host is connected to the deep learning processor and is used to transmit information data to the deep learning processor. The information data includes the preset necessary parameters, calculation instructions and data to be processed in the deep learning processor.

[0011] In one implementation of this application, the deep learning processor further includes:

[0012] The DMA (Direct Memory Access) section, the processor interrupt control section, and the data storage section.

[0013] Secondly, this application provides a method for applying a deep learning processor architecture system with an attention mechanism. The method, based on the deep learning processor architecture system with an attention mechanism, includes:

[0014] The deep learning processor obtains the preset necessary parameters, calculation instructions and data to be processed transmitted by the host.

[0015] Based on the preset necessary parameters, the processor configuration module completes the configuration of parameters for each module in the deep learning processor, the configuration of parameters for each unit in the attention mechanism control module, and the configuration of parameters for each module in the real-time calculation unit of attention mechanism parameters through the management information configuration module.

[0016] After configuring the parameters of each module in the real-time calculation unit for the attention mechanism parameters,

[0017] The input sequence module receives calculation instructions and data to be processed, and transmits the calculation instructions and data to be processed to the decomposition method reading module.

[0018] The decomposition method reading module determines the data decomposition format corresponding to the calculation instruction based on the preset input sequence decomposition method lookup table, and then obtains the decomposed data corresponding to the data to be processed; the decomposed data is sent to the attention parameter generation module, and the calculation instruction, decomposed data and data to be processed are sent to the input sequence buffer module;

[0019] The attention parameter generation module performs a preset linear transformation on the decomposed data to generate corresponding Q, K, and V vectors; the decomposed data and the corresponding Q, K, and V vectors are then sent to the attention score calculation and control FSM state machine.

[0020] The attention score calculation control FSM state machine controls the attention score calculation module according to preset steps to calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data; and sends the attention scores to the MUX data merging module.

[0021] The MUX data merging module obtains the data to be processed and the calculation instructions transmitted by the input sequence buffer module, aggregates and splices the data to be processed and the attention score, and sends the calculation instructions and the aggregation and splicing results to the optimal scheduling method selection and setting module.

[0022] The optimal scheduling method selection module verifies whether the attention score meets the preset verification rules. If it does, the scheduling mechanism to be used is determined according to the calculation instructions. The data packet containing the scheduling mechanism, the data to be processed, the attention score, and the calculation instructions is sent to the attention score output module.

[0023] According to the scheduling mechanism, the attention score output module sends the data packet to the corresponding implicit attention mechanism scheduling unit or explicit attention mechanism scheduling unit;

[0024] According to the preset allocation and scheduling strategy in the implicit attention mechanism scheduling unit or the explicit attention mechanism scheduling unit, the data to be processed and the corresponding attention score are sent to the computing unit control module in sequence.

[0025] The computing unit and deep learning algorithm in the computing unit control module are used to obtain the computing results and write them back to the host.

[0026] In one implementation of this application, the decomposition method reading module determines the data decomposition format corresponding to the calculation instruction based on a preset input sequence decomposition method lookup table, thereby obtaining the decomposed data corresponding to the data to be processed, specifically including:

[0027] A hash calculation is performed on the calculation instruction to obtain the storage address of the data decomposition format of the current calculation instruction in the preset input sequence decomposition method lookup table, and then the data decomposition format corresponding to the calculation instruction is read; the received data to be processed is formatted according to the data decomposition format to obtain the decomposed data.

[0028] In one implementation of this application, the attention score calculation module is controlled by the attention score calculation control FSM state machine according to preset steps to calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data, specifically including:

[0029] Based on the order of the input decomposed data, a preset linear transformation is performed on each decomposed data using the user target attention weight in the preset necessary parameters to generate the corresponding Q, K, and V vectors.

[0030] In one implementation of this application, the attention score calculation module is controlled by the attention score calculation control FSM state machine according to preset steps to calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data, specifically for:

[0031] The attention score is calculated for the Q, K, and V vectors corresponding to each input decomposed data, in a total of 4 steps:

[0032] S0: Receive the Q, K, and V vectors corresponding to the decomposed data;

[0033] S1. Transmit the Q, K, and V vectors corresponding to the decomposed data to the attention score calculation module, and control the target operator of the attention score calculation module to perform the dot product calculation of Q and K to obtain the result R1.

[0034] S2. The target operator of the attention score calculation module performs normalization calculation using the Softmax function to obtain the attention weight R2.

[0035] S3. The target operator of the control attention score calculation module performs a weighted summation calculation of attention weights. The attention weights are weighted and summed with the corresponding V vectors to obtain the weighted sum vector R3.

[0036] The data to be processed, R1, R2, and R3 are aggregated and concatenated one by one to form the attention score.

[0037] In one implementation of this application, the Q, K, and V vectors corresponding to the decomposed data are transmitted to the attention score calculation module, and the target operator of the attention score calculation module is controlled to perform the dot product calculation of Q and K to obtain the result R1, specifically including:

[0038] The similarity R1 is calculated using the formula: Where i represents the number of columns in vector Q, and j represents the number of rows in vector K;

[0039] The target operator of the attention score calculation module performs a Softmax function normalization calculation to obtain the attention weights R2, which specifically include:

[0040] Through the formula:

[0041] Normalization was performed; among them, Indicate the dimensions of Q and K;

[0042] Through the formula:

[0043] The attention weight R2 is calculated.

[0044] The target operator of the attention score calculation module performs a weighted summation of attention weights. The attention weights are weighted and summed with their corresponding V vectors to obtain a weighted sum vector R3, which specifically includes:

[0045] Through the formula:

[0046] This yields the weighted sum vector R3.

[0047] In one implementation of this application, according to a preset allocation scheduling strategy in the implicit attention mechanism scheduling unit or the explicit attention mechanism scheduling unit, the data to be processed and the corresponding attention score are sequentially sent to the computing unit control module, specifically including:

[0048] The implicit attention mechanism scheduling unit or the explicit attention mechanism scheduling unit sends the data to be processed and the corresponding attention score to the target computing unit in sequence according to the preset allocation scheduling strategy. The target computing unit then performs the corresponding deep learning data processing and calculation.

[0049] In one implementation of this application, the computational unit and deep learning algorithm in the computational unit control module are used to obtain the computational result, and the output result is written back to the host. Specifically, this includes:

[0050] After the target computing unit in the computing unit control board completes the computing task, the computing result is temporarily stored in the consistency cache of the computing unit control board, and then written back to the host terminal in the form of DMA through the PCIe interface (Peripheral Component Interconnect Express).

[0051] The CPU of the host terminal merges and processes the calculation results of the calculation instructions and the corresponding attention score allocation of each target calculation unit. According to the preset verification rules, it verifies the calculation results, and confirms that the calculation is completed when the preset verification rules are met.

[0052] As can be seen from the above technical solutions, this application has the following advantages:

[0053] I. Reduced the computing load on the host processor

[0054] The architecture described in this application decouples the attention score calculation function from the host processor to a dedicated hardware unit (the real-time calculation unit for attention mechanism parameters), allowing the host processor to handle only initial configuration management (through the management information configuration module). The collaborative operation of the attention score calculation control FSM state machine module and the calculation module achieves full hardware autonomy from input sequence decomposition and parameter generation to score calculation. This design eliminates the burden of frequent host processor intervention in data splitting, score calculation, and task scheduling found in traditional architectures, fundamentally avoiding critical process delays due to overload. The AXI bus topology directly connecting each functional module further reduces the data transport burden on the host processor.

[0055] Second, it improved the timeliness of data processing:

[0056] The integrated design of the attention mechanism control module and the computing unit control module constructs a closed-loop pipeline for computation-scheduling-execution. The linkage between the MUX data merging module and the optimal scheduling method selection module allows attention scores to be directly allocated to computing resources via implicit / explicit scheduling units, eliminating the need for cross-processor data transfers in traditional solutions. The introduction of the input sequence buffer module enables preloading of computing resources, which, combined with the parallel processing capabilities of the attention score calculation module, allows data to complete the entire process from feature extraction to result output on-chip.

[0057] Third, it reduces the complexity of system development:

[0058] The unified management provided by the processor configuration module (through the management information configuration module) abstracts the previously scattered operations such as attention mechanism configuration and computation unit parameter settings into a standardized instruction set. Developers only need to read the module's strategy parameters through the high-level API to automatically trigger subsequent attention parameter generation, score calculation, and scheduling allocation processes. The programmable interface of the explicit / implicit scheduling unit supports flexible switching between different algorithm paradigms. This "configure-as-you-go" feature lowers the technical threshold for developing deep learning systems, enabling researchers without hardware expertise to quickly deploy customized models. Attached Figure Description

[0059] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of the internal structure of a deep learning processor architecture system with an attention mechanism provided in an embodiment of this application.

[0061] Figure 2 This is a schematic diagram of the internal structure of an attention mechanism parameter real-time calculation unit provided in an embodiment of this application.

[0062] Figure 3 This is a schematic diagram of the internal structure of a deep learning processor architecture system provided in an embodiment of this application.

[0063] Figure 4 This is a detailed internal structure diagram of an efficient deep learning processor with an attention mechanism provided in an embodiment of this application.

[0064] Figure 5 This is a flowchart of a deep learning processor architecture system application method with attention mechanism provided in an embodiment of this application.

[0065] Explanation of key figure labels:

[0066] 100. Deep learning processor;

[0067] 110. Attention Mechanism Control Module; 111. Real-time Calculation Unit for Attention Mechanism Parameters; 1. Input Sequence Module; 2. Decomposition Method Reading Module; 3. Attention Parameter Generation Module; 4. Input Sequence Buffer Module; 5. Attention Score Calculation and Control FSM State Machine Module; 6. Attention Score Calculation Module; 7. MUX Data Merging Module; 8. Optimal Scheduling Method Selection and Setting Module; 9. Attention Score Output Module; 10. Management Information Configuration Module; 112. Implicit Attention Mechanism Scheduling Unit; 113. Explicit Attention Mechanism Scheduling Unit;

[0068] 120. Computing Unit Control Block; 130. Processor Configuration Block; 140. AXI Control / Data Bus; 150. On-Chip Cache Block; 160. DMA Block; 170. Processor Interrupt Control Block; 180. Data Storage Block; 200. Host Unit. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] Those skilled in the art should understand that the embodiments described below are merely preferred embodiments of this disclosure and do not imply that this disclosure can only be implemented through these preferred embodiments. These preferred embodiments are merely used to explain the technical principles of this disclosure and are not intended to limit the scope of protection of this disclosure. Based on the preferred embodiments provided by this disclosure, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of this disclosure.

[0071] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0072] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0073] This application Figure 1-2 As shown, this is a deep learning processor architecture system with an attention mechanism provided in an embodiment of this application. Figure 1-2 As shown, the system provided in this application embodiment mainly includes:

[0074] Deep learning processor 100, such as Figure 1 As shown, the deep learning processor 100 includes: an attention mechanism control module 110, a computing unit control module 120, a processor configuration module 130, and an AXI control / data bus 140 connecting the various modules; and the attention mechanism control module 110 and the computing unit control module 120 adopt an integrated design.

[0075] The attention mechanism control module 110 includes: a real-time calculation unit for attention mechanism parameters 111, an implicit attention mechanism scheduling unit 112, and an explicit attention mechanism scheduling unit 113.

[0076] like Figure 2As shown, the real-time attention mechanism parameter calculation unit 111 includes: an input sequence module 1, a decomposition method reading module 2 connected to the input sequence module 1, an attention parameter generation module 3 connected to the decomposition method reading module 2, an input sequence buffer module 4, an attention score calculation and control FSM state machine module 5 connected to the attention parameter generation module 3, an attention score calculation module 6 connected to the attention score calculation and control FSM state machine module 5, a MUX data merging module 7 connected to the input sequence buffer module 4 and the attention score calculation and control FSM state machine module 5, an optimal scheduling method selection and setting module 8 connected to the MUX data merging module 7, an attention score output module 9 connected to the optimal scheduling method selection and setting module 8, and a management information configuration module 10 connected to all modules; and the management information configuration module 10 is connected to the processor configuration module 130, and the attention score output module 9 is connected to the implicit attention mechanism scheduling unit 112 and the explicit attention mechanism scheduling unit 113.

[0077] To elaborate further, Figure 2 The function of the attention mechanism control module 110 is to receive all deep learning instructions to be executed and related data to be processed, calculate the attention scores corresponding to all data to be processed in sequence according to the input processing instruction category and related user configuration data, and package the input deep learning processing data and the corresponding attention scores, and send the data packets to the implicit attention mechanism scheduling unit 112 / explicit attention mechanism scheduling unit 113 in sequence according to the type best matched by the attention score calculation results.

[0078] from Figure 2 As can be seen, the architecture of the real-time calculation unit 111 for attention mechanism parameters includes a management information configuration module 10, an input sequence module 1, a decomposition method reading module 2, an input sequence caching module 4, an attention parameter generation module 3, an attention score calculation and control FSM state machine module 5, an attention score calculation module 6 (including a series of underlying calculation function units such as ALU, FPU, and SFU), an attention score output module 9, a MUX data merging module 7, and an optimal scheduling method selection and setting module 8.

[0079] Among them, the management information configuration module 10 is responsible for completing the necessary parameterization settings and initialization configurations for other modules, and configuring the information preset by the Host user; the input sequence module 1 is responsible for receiving the calculation instructions and related data to be processed from the on-chip cache module 150 in the order of the user's deep learning requests; the decomposition method reading module 2 is used to read the data decomposition format corresponding to the current instruction according to the data decomposition format storage address stored in the input sequence decomposition method lookup table, and to adjust the format of the received original data to be processed; the input sequence caching module 4 is used to temporarily store the instructions and the input data decomposition results after the format adjustment is completed; the attention parameter generation module 3 is responsible for performing a specific linear transformation on each input data according to the order of the input decomposition data and based on the user configuration information, thereby generating the Q, K, and V vectors corresponding to each group of data; attention The FSM state machine module 5, responsible for controlling the attention score calculation module 6 to perform attention score calculations on the Q, K, and V vectors corresponding to each input raw decomposed data according to preset steps, and integrating the calculation results; the MUX data merging module 7 is responsible for reading the split input raw data to be processed from the input sequence buffer module 4, and completing the data aggregation and splicing with the calculation results R1, R2, and R3 of the attention score calculation control FSM state machine module 5 in a one-to-one correspondence; the optimal scheduling method selection and setting module 8 is responsible for performing secondary verification on the raw decomposed data and the corresponding attention score calculation results (with preset verification rules, and those skilled in the art can determine the specific rule content and scope according to actual needs), and then judging whether the current task is more suitable to be assigned to subsequent calculation units using implicit attention mechanism scheduling or explicit attention mechanism scheduling according to the input instructions.

[0080] The Deep Learning Processor 100 architecture system can be further specified, such as... Figure 3 As shown, this application can be specifically described as follows:

[0081] Figure 3 The system demonstrates an efficient deep learning processor 100 architecture system with an attention mechanism. Server user nodes are connected to the host 200 via a network, and the heterogeneous deep learning processor 100 communicates and exchanges information with the host 200 via a PCIe interface in the form of DMA.

[0082] The Deep Learning Processor 100 mainly consists of a DMA data transfer module ( Figure 3 It is composed of several parts, including (not shown), attention mechanism control module 110, computing unit control module 120, on-chip cache module 150, etc.

[0083] After receiving a deep learning processing request from a user, the host 200 sends computation instructions and the image / text data to be processed to the deep learning processor 100 via the PCIe interface. The attention mechanism control module 110 of the deep learning processor 100 calculates the attention mechanism parameters for the relevant data of the current computation request and schedules the specific computation task to the target computation unit of the computation unit control module 120. After the computation is completed, the computation result is sent to the requesting user terminal via the host 200 through the PCIe interface in the form of DMA.

[0084] In addition, the deep learning processor 100 also includes:

[0085] DMA module 160, processor interrupt control module 170, data storage module 180.

[0086] Deep learning processors can be further specified as, for example Figure 4 As shown, a deep learning processor can specifically be:

[0087] Figure 4 The detailed architecture of a high-efficiency deep learning processor 100 with an attention mechanism is shown, including an attention mechanism control module 110 (containing a real-time calculation unit 111 for attention mechanism parameters, an implicit attention mechanism scheduling unit 112, and an explicit attention mechanism scheduling unit 113), a DMA module 160, a computing unit control module 120 (containing all computing units deployed in the high-efficiency deep learning processor 100), a processor interrupt control module 170, a processor configuration module 130, a data storage module 180, and an AXI control / data bus 140 connecting the various modules.

[0088] Among them, the processor interrupt control module 170 is responsible for handling interrupt handling requests from the host 200;

[0089] The processor configuration module 130 is responsible for configuring the relevant parameters of the Host 200 for the Attention Mechanism Control Module 110, the Computing Unit Control Module 120, and the Data Storage Module 180, including the Instruction TCM (Tightly Coupled Memory) and Data TCM (Tightly Coupled Memory Data).

[0090] The DMA module 160 deploys a DMA data transceiver engine, which is responsible for data exchange between the processor and the host memory, and for sending and receiving computing task-related information and transmitting computing results in real time.

[0091] The attention mechanism control module 110 is equipped with a real-time attention mechanism parameter calculation unit 111, an input sequence decomposition method lookup table (which stores all user instructions and the decomposition format of the data to be processed corresponding to the instructions), an implicit attention mechanism scheduling unit 112, and an explicit attention mechanism scheduling unit 113. This module is responsible for calculating the attention mechanism parameters of the relevant data to be processed for the current user's deep learning computation request and scheduling the specific computation task containing the attention mechanism parameters to the target computation unit of the computation unit control module 120. The attention mechanism control module 110 and the computation unit control module 120 adopt an integrated design. After the attention mechanism parameters of the data to be processed are calculated, the task scheduling can be directly performed on the original data without going through the cumbersome and time-consuming system bus transmission.

[0092] The computing unit control module 120 deploys all the deep learning computing units of the processor and is responsible for executing deep learning computing tasks on the data to be processed, including attention mechanism parameters. The deep learning computing units include scalar data computing functions, vector data computing functions, tensor data computing functions, etc., which are necessary for deep learning data processing. The computing unit control module 120 is responsible for the unified coordination and control of all computing units.

[0093] Data storage module 180 includes instruction TCM storage and data TCM storage, which are responsible for storing relevant instruction data and calculation raw input data;

[0094] Different modules are connected via the AXI control bus and the AXI data bus. The AXI control / data bus 140 provides the necessary instructions and related data for operation. All functional modules are uniformly controlled and deployed by the high-efficiency deep learning processor 100 with an attention mechanism.

[0095] In addition, the embodiments provide a deep learning processor architecture system with an attention mechanism, such as Figure 5 As shown in the embodiments of this application, the method mainly includes the following steps:

[0096] Step 210: Obtain the preset necessary parameters, calculation instructions, and data to be processed transmitted by the host through the deep learning processor. Based on the preset necessary parameters, the processor configuration module completes the configuration of parameters for each module in the deep learning processor, the configuration of parameters for each unit in the attention mechanism control module, and the configuration of parameters for each module in the real-time calculation unit of the attention mechanism parameters through the management information configuration module.

[0097] Those skilled in the art will understand that configuring the parameters of each module in the real-time calculation unit for attention mechanism parameters can specifically be achieved by the management information configuration module completing the necessary parameterization settings (the necessary parameters are preset necessary parameters) for the input sequence module, decomposition method reading module, input sequence caching module, attention parameter generation module, attention score calculation and control FSM state machine module, attention score calculation module, optimal scheduling method selection and setting module, attention score output, and other modules.

[0098] Step 220: After configuring the parameters of each module in the real-time calculation unit of the attention mechanism parameters, the calculation instructions and the data to be processed are received through the input sequence module, and the calculation instructions and the data to be processed are transmitted to the decomposition method reading module.

[0099] It should be noted that the above data transmission method is as follows: the calculation instructions and the data to be processed are sent to the decomposition method reading module respectively according to the input order.

[0100] Step 230: The decomposition method reading module determines the data decomposition format corresponding to the calculation instruction according to the preset input sequence decomposition method lookup table, and then obtains the decomposed data corresponding to the data to be processed; the decomposed data is sent to the attention parameter generation module, and the calculation instruction, decomposed data and data to be processed are sent to the input sequence cache module.

[0101] The decomposition method reading module determines the data decomposition format corresponding to the calculation instruction based on a preset input sequence decomposition method lookup table, thereby obtaining the decomposed data corresponding to the data to be processed. Specifically, this can be:

[0102] The calculation instruction is hashed to obtain the storage address of the data decomposition format of the current calculation instruction in the preset input sequence decomposition method lookup table, and then the data decomposition format corresponding to the calculation instruction is read. Subsequently, the decomposition method reading module adjusts the format of the received data to be processed according to the data decomposition format (including the decomposition data bit width, decomposition format scalar, vector, matrix attributes, decomposition packet header and packet tail signals, etc.).

[0103] Step 240: Perform a preset linear transformation on the decomposed data through the attention parameter generation module to generate the corresponding Q, K, and V vectors; send the decomposed data and the corresponding Q, K, and V vectors to the attention score calculation and control FSM state machine.

[0104] Specifically, the attention score calculation module is controlled by the FSM state machine according to preset steps to calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data. This can be achieved by:

[0105] Based on the order of the input decomposed data, a preset linear transformation is performed on each decomposed data using the user target attention weight in the preset necessary parameters to generate the corresponding Q, K, and V vectors.

[0106] It should be noted that the Q, K, and V vectors are the three elements of attention parameter calculation: Query, Key, and Value vectors.

[0107] Step 250: Control the FSM state machine to control the attention score calculation module according to the preset steps through attention score calculation, calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data, and send the attention scores to the MUX data merging module.

[0108] The attention score calculation control FSM state machine controls the attention score calculation module according to preset steps to calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data. Specifically, it is used for:

[0109] The attention score is calculated for the Q, K, and V vectors corresponding to each input decomposed data, in a total of 4 steps:

[0110] S0: Receive the Q, K, and V vectors corresponding to the decomposed data;

[0111] S1. Transmit the Q, K, and V vectors corresponding to the decomposed data to the attention score calculation module, and control the target operator (ALU, FPU... the specific operator category is related to the data category) of the attention score calculation module to perform the dot product calculation of Q and K to obtain the result R1;

[0112] S2. The target operator of the attention score calculation module performs normalization calculation using the Softmax function to obtain the attention weight R2.

[0113] S3. The target operator of the control attention score calculation module performs a weighted summation calculation of attention weights. The attention weights are weighted and summed with the corresponding V vectors to obtain the weighted sum vector R3.

[0114] The data to be processed, R1, R2, and R3 are aggregated and concatenated one by one to form the attention score.

[0115] More specifically, the Q, K, and V vectors corresponding to the decomposed data are transmitted to the attention score calculation module, which controls the target operator of the attention score calculation module to perform the dot product calculation of Q and K to obtain the result R1, which can be specifically as follows:

[0116] The similarity R1 is calculated using the formula: Where i represents the number of columns in vector Q, and j represents the number of rows in vector K;

[0117] The target operator of the attention score calculation module performs a Softmax function normalization calculation to obtain the attention weight R2, which can be specifically:

[0118] Through the formula:

[0119] Normalization was performed; among them, Indicate the dimensions of Q and K;

[0120] Through the formula:

[0121] The attention weight R2 is calculated.

[0122] The target operator of the attention score calculation module performs a weighted summation of attention weights. The attention weights are summed with their corresponding V vectors to obtain a weighted sum vector R3, which can be specifically:

[0123] Through the formula:

[0124] This yields the weighted sum vector R3.

[0125] Step 260: Obtain the data to be processed and the calculation instructions transmitted by the input sequence buffer module through the MUX data merging module, aggregate and splice the data to be processed and the attention score, and send the calculation instructions and the aggregation and splicing results to the optimal scheduling method selection and setting module.

[0126] Step 270: Verify whether the attention score meets the preset verification rules through the optimal scheduling method selection and setting module. If it does, determine the scheduling mechanism to be used according to the calculation instructions; send the data packet containing the scheduling mechanism, the data to be processed, the attention score, and the calculation instructions to the attention score output module.

[0127] Step 280: According to the scheduling mechanism, the attention score output module sends the data packet to the corresponding implicit attention mechanism scheduling unit or explicit attention mechanism scheduling unit; according to the preset allocation scheduling strategy in the implicit attention mechanism scheduling unit or explicit attention mechanism scheduling unit, the data to be processed and the corresponding attention score are sent to the computing unit control module in sequence.

[0128] According to the preset allocation and scheduling strategy in the implicit attention mechanism scheduling unit or the explicit attention mechanism scheduling unit, the data to be processed and the corresponding attention score are sent sequentially to the computing unit control module, specifically including:

[0129] The implicit attention mechanism scheduling unit or the explicit attention mechanism scheduling unit sends the data to be processed and the corresponding attention score to the target computing unit in sequence according to the preset allocation scheduling strategy. The target computing unit then performs the corresponding deep learning data processing and calculation.

[0130] Step 290: Use the computing units and deep learning algorithms in the computing unit control module to obtain the computing results and write the computing results back to the host.

[0131] The computation unit controls the computation units and deep learning algorithms within the computation unit control module to obtain computation results, and then writes the output results back to the host machine. Specifically, this includes:

[0132] After the target computing unit in the computing unit control module completes the computing task, the computing result is temporarily stored in the consistency cache of the computing unit control module, and then written back to the host terminal in the form of DMA through the PCIe interface.

[0133] The CPU of the host terminal merges and processes the calculation results of the calculation instructions and the corresponding attention score allocation of each target calculation unit. According to the preset verification rules, it verifies the calculation results, and confirms that the calculation is completed when the preset verification rules are met.

[0134] Based on the above description, as an example, this embodiment can be:

[0135] 1. Users send deep learning computing or application-related requests to the host.

[0136] 2. The host CPU receives the current request instruction and the image / text data to be processed, and then sends the instruction and related data to the deep learning processor with attention mechanism via PCIe interface in the form of DMA.

[0137] 3. The attention mechanism control module of the deep learning processor includes an attention mechanism parameter calculation unit that calculates the attention score corresponding to all the data to be processed in sequence according to the input processing instructions and relevant user configuration data.

[0138] 4. After calculating the attention score, the attention mechanism parameter calculation unit packages the input quasi-deep learning processing data and the corresponding attention score. Based on the best-fit type (implicit / explicit) of the attention score calculation result, the data packet is sent sequentially to the implicit attention mechanism scheduling unit / explicit attention mechanism scheduling unit to generate the corresponding allocation and scheduling strategy.

[0139] 5. The implicit attention mechanism scheduling unit / explicit attention mechanism scheduling unit sends the input data to be processed and the corresponding attention score to the target computing unit in sequence according to the generated allocation and scheduling strategy. The target computing unit then performs the corresponding deep learning data processing and calculation.

[0140] 6. After the target computing unit completes the computing task, the computing results are temporarily stored in the consistency cache of the computing unit's top-level control board, and then written back to the host terminal in the form of DMA through the PCIe interface.

[0141] 7. The CPU merges the current instruction data and the calculation results of each execution computing unit corresponding to the attention score allocation, performs a second check on the deep learning calculation results, and confirms that the calculation is complete.

[0142] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A deep learning processor architecture system with an attention mechanism, characterized in that, The system includes: The deep learning processor includes: an attention mechanism control module, a computing unit control module, a processor configuration module, and an AXI control / data bus connecting the various modules; and the attention mechanism control module and the computing unit control module adopt an integrated design. The attention mechanism control module includes: a real-time calculation unit for attention mechanism parameters, an implicit attention mechanism scheduling unit, and an explicit attention mechanism scheduling unit. The real-time attention mechanism parameter calculation unit includes: an input sequence module, a decomposition method reading module connected to the input sequence module, an attention parameter generation module and an input sequence buffer module connected to the decomposition method reading module, an attention score calculation and control FSM state machine module connected to the attention parameter generation module, an attention score calculation module connected to the attention score calculation and control FSM state machine module, a MUX data merging module connected to the input sequence buffer module and the attention score calculation and control FSM state machine module, an optimal scheduling method selection and setting module connected to the MUX data merging module, an attention score output module connected to the optimal scheduling method selection and setting module, and a management information configuration module connected to all modules; the management information configuration module is connected to the processor configuration module, and the attention score output module is connected to the implicit attention mechanism scheduling unit and the explicit attention mechanism scheduling unit.

2. The deep learning processor architecture system with attention mechanism according to claim 1, characterized in that, The system also includes: The host is connected to the deep learning processor and is used to transmit information data to the on-chip cache of the deep learning processor. The information data includes the preset necessary parameters, calculation instructions and data to be processed in the deep learning processor.

3. The deep learning processor architecture system with attention mechanism according to claim 1, characterized in that, The deep learning processor also includes: DMA module, processor interrupt control module, data storage module.

4. A method for applying a deep learning processor architecture system with an attention mechanism, based on the deep learning processor architecture system with an attention mechanism as described in claim 1, characterized in that, The method includes: The deep learning processor obtains the preset necessary parameters, calculation instructions and data to be processed transmitted by the host. Based on the preset necessary parameters, the processor configuration module completes the configuration of parameters for each module in the deep learning processor, the configuration of parameters for each unit in the attention mechanism control module, and the configuration of parameters for each module in the real-time calculation unit of attention mechanism parameters through the management information configuration module. After configuring the parameters of each module in the real-time calculation unit of the attention mechanism parameters, the calculation instructions and data to be processed are received through the input sequence module, and the calculation instructions and data to be processed are transmitted to the decomposition method reading module. The decomposition method reading module determines the data decomposition format corresponding to the calculation instruction based on the preset input sequence decomposition method lookup table, and then obtains the decomposed data corresponding to the data to be processed; the decomposed data is sent to the attention parameter generation module, and the calculation instruction, decomposed data and data to be processed are sent to the input sequence buffer module; The attention parameter generation module performs a preset linear transformation on the decomposed data to generate corresponding Q, K, and V vectors; the decomposed data and the corresponding Q, K, and V vectors are then sent to the attention score calculation and control FSM state machine. The attention score calculation control FSM state machine controls the attention score calculation module according to preset steps to calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data; and sends the attention scores to the MUX data merging module. The MUX data merging module obtains the data to be processed and the calculation instructions transmitted by the input sequence buffer module, aggregates and splices the data to be processed and the attention score, and sends the calculation instructions and the aggregation and splicing results to the optimal scheduling method selection and setting module. The optimal scheduling method selection module verifies whether the attention score meets the preset verification rules. If it does, the scheduling mechanism to be used is determined according to the calculation instructions. The data packet containing the scheduling mechanism, the data to be processed, the attention score, and the calculation instructions is sent to the attention score output module. According to the scheduling mechanism, the attention score output module sends the data packet to the corresponding implicit attention mechanism scheduling unit or explicit attention mechanism scheduling unit; according to the preset allocation scheduling strategy in the implicit attention mechanism scheduling unit or explicit attention mechanism scheduling unit, the data to be processed and the corresponding attention score are sent to the computing unit control module in sequence. The computing unit and deep learning algorithm in the computing unit control module are used to obtain the computing results and write them back to the host.

5. The application method of the deep learning processor architecture system with attention mechanism according to claim 4, characterized in that, The decomposition method reading module determines the data decomposition format corresponding to the calculation instruction based on a preset input sequence decomposition method lookup table, thereby obtaining the decomposed data corresponding to the data to be processed, specifically including: A hash calculation is performed on the calculation instruction to obtain the storage address of the data decomposition format of the current calculation instruction in the preset input sequence decomposition method lookup table, and then the data decomposition format corresponding to the calculation instruction is read; the received data to be processed is formatted according to the data decomposition format to obtain the decomposed data.

6. The application method of the deep learning processor architecture system with attention mechanism according to claim 4, characterized in that, The attention score calculation module is controlled by the FSM state machine according to preset steps to calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data. Specifically, this includes: Based on the order of the input decomposed data, a preset linear transformation is performed on each decomposed data using the user target attention weight in the preset necessary parameters to generate the corresponding Q, K, and V vectors.

7. The application method of the deep learning processor architecture system with attention mechanism according to claim 4, characterized in that, The attention score calculation control FSM state machine controls the attention score calculation module according to preset steps to calculate the attention scores of the Q, K, and V vectors corresponding to the decomposed data. Specifically, it is used for: The attention score is calculated for the Q, K, and V vectors corresponding to each input decomposed data, in a total of 4 steps: S0: Receive the Q, K, and V vectors corresponding to the decomposed data; S1. Transmit the Q, K, and V vectors corresponding to the decomposed data to the attention score calculation module, and control the target operator of the attention score calculation module to perform the dot product calculation of Q and K to obtain the result R1. S2. The target operator of the attention score calculation module performs normalization calculation using the Softmax function to obtain the attention weight R2. S3. The target operator of the control attention score calculation module performs a weighted summation calculation of attention weights. The attention weights are weighted and summed with the corresponding V vectors to obtain the weighted sum vector R3. The data to be processed, R1, R2, and R3 are aggregated and concatenated one by one to form the attention score.

8. The application method of the deep learning processor architecture system with attention mechanism according to claim 7, characterized in that, The Q, K, and V vectors corresponding to the decomposed data are transmitted to the attention score calculation module. The target operator of the attention score calculation module is controlled to perform the dot product calculation of Q and K to obtain the result R1, which specifically includes: The similarity R1 is calculated using the formula: Where i represents the number of columns in vector Q, and j represents the number of rows in vector K; The target operator of the attention score calculation module performs a Softmax function normalization calculation to obtain the attention weights R2, which specifically include: Through the formula: Normalization was performed; among them, Indicate the dimensions of Q and K; Through the formula: The attention weight R2 is calculated. The target operator of the attention score calculation module performs a weighted summation of attention weights. The attention weights are weighted and summed with their corresponding V vectors to obtain a weighted sum vector R3, which specifically includes: Through the formula: This yields the weighted sum vector R3.

9. The application method of the deep learning processor architecture system with attention mechanism according to claim 4, characterized in that, According to the preset allocation and scheduling strategy in the implicit attention mechanism scheduling unit or the explicit attention mechanism scheduling unit, the data to be processed and the corresponding attention score are sent sequentially to the computing unit control module, specifically including: The implicit attention mechanism scheduling unit or the explicit attention mechanism scheduling unit sends the data to be processed and the corresponding attention score to the target computing unit in sequence according to the preset allocation scheduling strategy. The target computing unit then performs the corresponding deep learning data processing and calculation.

10. The application method of the deep learning processor architecture system with attention mechanism according to claim 4, characterized in that, The computational unit controls the computational units and deep learning algorithms within the computational unit control module to obtain computational results, and then writes the output results back to the host machine. Specifically, this includes: After the target computing unit in the computing unit control module completes the computing task, the computing result is temporarily stored in the consistency cache of the computing unit control module, and then written back to the host terminal in the form of DMA through the PCIe interface. The CPU of the host terminal merges and processes the calculation results of the calculation instructions and the corresponding attention score allocation of each target calculation unit. According to the preset verification rules, it verifies the calculation results, and confirms that the calculation is completed when the preset verification rules are met.

Images