An FPGA-based visual self-attention accelerator optimization method

Abstract

The application discloses a visual self-attention accelerator optimization method based on FPGA, and comprises the following steps: dynamic token pruning is performed on a visual self-attention model through a dynamic token pruning scheme, redundant information is removed, and the calculation amount of the visual self-attention model is reduced; the calculation process is divided through the design mode of a single visual self-attention calculation layer on the FPGA and the use of a matrix cutting mode, the optimal calculation resource allocation strategy is solved based on a genetic algorithm, and maximum load balancing is realized; the application reduces the calculation amount, reduces the model running time, and improves the running efficiency of the accelerator.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic information, and more particularly, to a visual self-attention accelerator optimization method based on FPGA. BACKGROUND

[0002] Compared with the traditional convolutional neural network (CNN) network, the visual self-attention model has a large number of parameters and a large amount of calculation per inference. The increase in the number of model parameters and the amount of calculation puts higher requirements on the performance of the computing unit and the bandwidth of the FPGA design. Therefore, how to compress the existing model and efficiently utilize the on-chip computing resources is an important problem in the design of the accelerator.

[0003] In terms of software algorithms, the token information in the visual task contains a large amount of redundancy. Exploring and removing the redundant information can greatly reduce the amount of calculation of the visual self-attention. According to the optimal token pruning ratio selected for the redundancy of each layer of tokens, the model performance can be maintained while the amount of calculation is reduced.

[0004] In terms of hardware accelerator design, there are multiple "intermediate calculation result-intermediate calculation result" multiplications and layer normalization operations in the visual self-attention model, which means that the calculation cannot share a set of computing units with the operators in the convolutional neural network (CNN), i.e., the model parameters cannot be rearranged and stored in the accelerator for calculation with the intermediate results.

[0005] Therefore, in the design of the visual self-attention model accelerator, different computing units need to be designed for different computing paradigms. In addition, a suitable data chunking scheme needs to be selected to achieve efficient computing efficiency of the FPGA. Token pruning may cause the computing efficiency of the accelerator to decrease when running different layers of the model, and the effect of token pruning on the running efficiency of the accelerator also needs to be considered in the design of the accelerator. SUMMARY

[0006] To solve the problem of low running efficiency of the accelerator caused by the large amount of calculation of the visual self-attention model in the prior art, the present application proposes a visual self-attention accelerator optimization method based on FPGA, which reduces the amount of calculation, reduces the model running time, and improves the running efficiency of the accelerator.

[0007] To solve the above technical problems, the technical solution adopted by the present application is as follows:

[0008] A visual self-attention accelerator optimization method based on FPGA, comprising the following steps:

[0009] The dynamic token pruning scheme is used to prune the dynamic token of the visual self-attention model, remove redundant information, and obtain an optimized visual self-attention model.

[0010] When the visual self-attention model accelerator is constructed, the single visual self-attention calculation layer on the FPGA is designed, the calculation process is divided by using the matrix cutting method, the optimal calculation resource allocation strategy is solved based on the genetic algorithm, the load balancing is maximized, and the optimized visual self-attention model accelerator is obtained.

[0011] The optimized visual self-attention model is run by the optimized visual self-attention model accelerator.

[0012] Preferably, the dynamic token pruning scheme is as follows:

[0013] According to the output policy network in the reinforcement learning framework, the corresponding token pruning ratio of each layer in the visual self-attention model is predicted.

[0014] The visual self-attention model sorts each token according to the pruning ratio, and removes the token with the smallest importance.

[0015] The pruning behavior of the current layer is expressed as a compression ratio, and the loss of the corresponding visual self-attention model under the pruning behavior is used as the reward input of the scoring network, so as to update the integral network.

[0016] The token pruning of the next layer of the visual self-attention model is performed until the pruning of all layers is completed.

[0017] Further, for the dynamic token pruning scheme, an asynchronous advantage action evaluation algorithm is used to model the reward function and the optimal strategy of the agent.

[0018] The optimal strategy is the best pruning ratio of each layer.

[0019] The framework of reinforcement learning is composed of an environment and an agent; the environment is responsible for transmitting the observation value and the reward value of each state to the agent; the agent receives the observation value and the reward value of each state transmitted by the environment for its own training, and performs task allocation of the visual self-attention model according to the received state variable.

[0020] In the reinforcement learning environment, the state vector is returned after each pruning step, and the agent judges the change of the environment and makes the next decision.

[0021] For an N-layer visual self-attention model, an N+1-dimensional vector is used to represent the state of the visual self-attention model.

[0022] When the agent runs to the i-th step, i.e. prunes to the i-th layer, the state vector is defined as follows:

[0023]

[0024] where i / n is the agent's indication of the current pruning layer, scaled to the range [0,1];RTi represents the proportion of tokens remaining at the i-th layer.

[0025] Further, the reward function scores the pruning effect of the agent and trains the agent's parameters as part of the environmental feedback, optimizing the agent's behavior.

[0026] The mathematical form of the reward function is as follows:

[0027] Reward=λ×[Total_PR-α×acc_loss 2 ]

[0028] The reward function contains two parts of compression ratio and accuracy loss, and contains two adjustable parameters λ and α;The λ factor is the scaling factor of the overall reward function, which is used to scale the reward function value to [0,1];The α is an importance adjustment factor, which is used to adjust the importance between the compression ratio and the accuracy loss, and then get different compression ratios.

[0029] The greater the α factor in the implementation process indicates that the agent tends to maintain the reduced accuracy loss, and vice versa, it tends to accept the accuracy loss to obtain a high compression return.

[0030] Preferably, the accelerator comprises an attention calculation subsystem, a forward network calculation subsystem, an AXI read-write command control module, a first selector, and a weight cache module.

[0031] The attention calculation subsystem comprises a first input buffer module, a first addition module, a first normalization layer, a second selector, a multi-head attention module, and a sorting module.

[0032] The forward network calculation subsystem comprises a second input buffer module, a second addition module, a second normalization layer, and a forward network module.

[0033] After the AXI read-write command control module receives the data of the crossbar matrix, it is respectively sent to the attention calculation subsystem, the forward network calculation subsystem, and the weight cache module.

[0034] After the weight cache module receives the data of the AXI read-write command control module, it caches the weight data required by the attention calculation subsystem and the forward network calculation subsystem, and respectively sends the corresponding weight data to the attention calculation subsystem and the forward network calculation subsystem.

[0035] The first input buffer module is used for buffering data read from the AXI read-write command control module, and the buffered data is sent to the first addition module; the first addition module sends the data after addition operation to the first normalization layer, the second selector and the first selector; the first normalization layer normalizes the data output by the first addition module and sends it to the second selector; the second selector selects the data input to the multi-head attention module at the current moment, and sends the selected data to the multi-head attention module;

[0036] The multi-head attention module receives the selected data sent by the second selector and the weight data sent by the weight buffer module, and performs multi-head attention mechanism calculation; the sorting module sorts and indexes according to the importance scores of each token to complete the pruning of unimportant tokens, and outputs the sorted data to the first selector.

[0037] The first selector selects the data input to the AXI read-write command control module at the current moment, and sends the selected data to the AXI read-write command control module.

[0038] The second input buffer module is used for buffering data read from the AXI read-write command control module, and the buffered data is sent to the second addition module; the second addition module sends the data after addition operation to the second normalization layer; the second normalization layer normalizes the data output by the second addition module and sends it to the forward network module and the AXI read-write command control module; the forward network module performs forward inference operation to obtain the calculation result of the current layer.

[0039] Further, assuming that the time required for the forward network calculation subsystem and the attention calculation subsystem to complete one operation is ΔTF and ΔTA, the average time required for the accelerator to complete one layer operation is ΔT.

[0040] ΔT=max(ΔTF,△TA);

[0041] Let Q_space be the overall exploration space of hardware design parameters, and Q_space_h be a set of design parameters that maximize the calculation efficiency of the accelerator.

[0042] Q_space_h=argminQ|ΔTF-ΔTA|;

[0043] The calculation of the multi-head attention module consists of attention mechanism calculation, splicing operation and matrix multiplication. When operating, the input vector matrix X passes through three matrix multiplication units to obtain feature vector matrices Q, K and V respectively; the feature vector matrix Q is multiplied by the transposed feature vector matrix K to obtain the importance scores between tokens through a normalization exponential function, and then multiplied by the feature vector matrix V to obtain the final output of the single-head attention mechanism.

[0044] ΔTQ, ΔTK, ΔTV, ΔTQK, ΔTMM are the calculation time of each matrix multiplication operation of the attention mechanism respectively.

[0045] Then we have:

[0046] ΔTA= max(ΔTQ, ΔTK, ΔTV, ΔTQK, ΔTMM)

[0047] Qsh= argminQ max(ΔTQ, ΔTK, ΔTV, ΔTQK, ΔTMM)

[0048] Qsh is the parameter set that minimizes the processing time ΔTA.

[0049] TQK is the multiplication of the feature vector matrix Q and the transpose of the feature vector matrix K.

[0050] TMM is the multiplication of the result of Q*K^T and V.

[0051] Further, the multi-head attention module further comprises a feature vector calculation module.

[0052] When calculating the feature vector, the matrix slicing method is used to complete a matrix operation in a loop.

[0053] The matrix slicing method is as follows:

[0054] The input token matrix and the parameter matrix are cut into small matrices with dimensions [xh, xw] and [xw, ww] respectively; each small matrix is used as a basic calculation unit; each PE unit is composed of xw multipliers and a level addition tree, and a register is inserted between each multiplier and adder.

[0055] The entire PE array is composed of xh×ww PE units, and when the calculation array processes a continuous data stream, it completes a small matrix multiplication in an average clock cycle.

[0056] In the Q×KT attention score module design, the input query Query and key Key block matrix will first be rearranged into the slicing size [xhqk, xwqk] and [xwqk, wwqk] of the module.

[0057] Pong cache is used to cache the data of the previous layer, reducing the calculation unit stagnation caused by matrix switching; the normalized exponential function input matrix is input into xhi×wwi natural exponential evaluation units by row respectively, and the calculation result is input into the summation array to calculate the sum of the natural exponential of each row of data.

[0058] The natural exponential result is written into the cache after rearrangement, and when the sum of each row of natural exponential is calculated, it is input into the division array to further complete the operation of the final result.

[0059] The Matrix Mul module is responsible for multiplication of two matrices; for implementation of the multi-head attention mechanism, a part of the attention head is implemented and the final calculation result is obtained by accumulating the intermediate result.

[0060] Further, the sorting module comprises a sorting submodule and an index submodule.

[0061] The sorting submodule comprises a first counter, a double-tuned sequence generation module and a double-tuned sequence sorting unit.

[0062] The index submodule comprises a second counter, a third counter and a data-address index unit.

[0063] Further, the first counter counts the number of input tokens and outputs the token serial number corresponding to each token to the double-tuned sequence generation module.

[0064] The double-tuned sequence generation module generates a standard double-tuned sequence and sends it to the double-tuned sequence sorting unit for sorting.

[0065] Only the token importance score is sorted in the sorting process, and if an exchange occurs in the sorting process, the token importance score and the token serial number are exchanged together.

[0066] The operation of the index submodule comprises writing and reading.

[0067] The writing process is as follows:

[0068] The second counter selects the tokens output by the sorting part according to the pre-configured pruning retention number K, writes the serial numbers of the top K tokens with the highest importance into the data-address index unit, and counts the token labels according to the token end signal token_in_end when the token calculation result is input, and outputs the token labels; at this time, the token labels are used to query the data-address index unit to determine whether the token belongs to the token that needs to be retained; if the token needs to be retained, the calculation result of the token will be written into the off-chip DDR, otherwise it will not participate in subsequent operations.

[0069] The reading process is as follows:

[0070] According to the token label output by the third counter, the CAM is queried to read whether the storage corresponding to the address is 0 or 1, if it is 1, the calculation result of the token is retained, and the calculation result is stored in the DDR; if it is 0, it is not stored and does not participate in subsequent operations.

[0071] Further, the hardware design space is explored based on the genetic algorithm, and the corresponding target function is:

[0072] Qh_param = argmin Q max (ATF, AT A) = argmin Q AT

[0073] Wherein Qh_param is the optimal design parameter, Q is the search space of the overall design parameter.

[0074] According to the on-chip resource condition of FPGA, there are different overall resource constraints, that is, the on-chip DSP, storage and bandwidth of hardware design cannot exceed the highest value of the device; the calculation resource utilization is used as the performance measurement index of the scheme.

[0075] The calculation resource utilization is embodied as the ratio of the effective running time of the DSP unit to the overall running time.

[0076] Considering the influence of different layer running time, for the N-layer ViT network, the DSE target function is modified as:

[0077] Qh = argmin Q N ∑ i = 1 (ATi)

[0078] Through genetic algorithm search, the matrix block parameter of each matrix calculation module of the accelerator is obtained, and then the optimal hardware accelerator design scheme is obtained.

[0079] Compared with the prior art, the beneficial effects of the present application are:

[0080] 1. Through the dynamic token pruning scheme, the calculation amount is reduced under the condition of ensuring the model calculation performance, and then the model running time is reduced.

[0081] 2. Through the matrix block method, the calculation process is divided, the optimal calculation resource allocation strategy is solved based on genetic algorithm, the maximum load balancing effect is realized, and the running efficiency of the accelerator is improved. BRIEF DESCRIPTION OF DRAWINGS

[0082] Figure 1 The schematic diagram of the FPGA-based visual self-attention accelerator optimization method provided for the embodiment.

[0083] Figure 2 The schematic diagram of the dynamic token pruning scheme provided for the embodiment.

[0084] Figure 3 The structural block diagram of the accelerator provided for the embodiment.

[0085] Figure 4 The schematic diagram of the matrix block calculation provided for the embodiment.

[0086] Figure 5 The structural schematic diagram of the feature vector Q generation module provided for the embodiment.

[0087] Figure 6 A schematic diagram of the attention score module provided in the embodiment.

[0088] Figure 7 A schematic diagram of the accelerator runtime sequence provided for an embodiment. Detailed Implementation

[0089] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0090] Example 1

[0091] In this embodiment, as Figure 1 As shown, an optimization method for a visual self-attention accelerator based on FPGA includes the following steps:

[0092] The visual self-attention model is dynamically pruned using a dynamic token pruning scheme to remove redundant information, reduce the computational load of the visual self-attention model, and obtain an optimized visual self-attention model.

[0093] When constructing a visual self-attention model accelerator, the computation process is divided into segments by designing a single visual self-attention computation layer on an FPGA and using matrix partitioning. The optimal computational resource allocation strategy is solved based on a genetic algorithm to maximize load balancing, resulting in an optimized visual self-attention model accelerator.

[0094] Run the optimized visual self-attention model through the optimized visual self-attention model accelerator.

[0095] like Figure 2 As shown, the dynamic token pruning scheme is as follows:

[0096] Based on the output policy network in the reinforcement learning framework, predict the corresponding token pruning ratio (PR) for each layer in the visual self-attention model.

[0097] The output policy network provides a pruning ratio of 0.2, and there are a total of TN0 = 10 (tokens TokenNumber, TN) tokens. The number of tokens that need to be pruned at this layer is PR × TN = 2. At this time, the two smallest tokens out of the ten tokens are selected for pruning.

[0098] The visual self-attention model sorts the tokens according to the pruning ratio and discards the tokens with the least importance.

[0099] The pruning behavior of the current layer is represented as a compression ratio, and the loss of the corresponding visual self-attention model under this pruning behavior in terms of accuracy is used as the reward input of the scoring network to update the integral network.

[0100] Token pruning for the next layer of the visual self-attention model is performed until pruning for all layers is completed.

[0101] In this embodiment, for the dynamic token pruning scheme, an asynchronous advantage actor-critic algorithm is used to model the reward function and the optimal policy of the agent, respectively.

[0102] The optimal policy is the best pruning ratio for each layer.

[0103] The framework of reinforcement learning consists of an environment and an agent; the environment is responsible for delivering the observation values and reward values in each state to the agent; the agent receives the observation values and reward values in each state delivered by the environment for its own training, and performs task allocation for the visual self-attention model according to the received state variables.

[0104] In the reinforcement learning environment, the state vector is returned after each step of pruning is completed, and the agent uses it to determine the changes in the environment and make the next decision.

[0105] For an N-layer visual self-attention model, an N+1-dimensional vector is used to represent the state of the visual self-attention model.

[0106] When the agent runs to the i-th step, i.e., prunes to the i-th layer, the state vector is defined as follows:

[0107]

[0108] where i / n is intended to indicate the current pruning layer number of the agent, which is scaled to the range [0, 1]; RTi represents the remaining token ratio of the i-th layer.

[0109] More specifically, the reward function scores the pruning effect of the agent and is used as part of the environment feedback to train the agent parameters, optimizing the behavior of the agent.

[0110] The mathematical form of the reward function is as follows:

[0111] Reward=λ×[Total_PR-α×acc_loss 2 ]

[0112] The reward function includes the compression ratio and the accuracy loss, and also includes two adjustable parameters λ and α; where λ is the scaling factor of the overall reward function, which scales the reward function value to the range [0, 1]; α is an importance adjustment factor that adjusts the importance between the compression ratio and the accuracy loss, and thus obtains different compression ratios.

[0113] In the implementation process, the larger the α factor indicates that the agent tends to maintain a lower accuracy loss, and vice versa.

[0114] Embodiment 2

[0115] In this embodiment, as shown in the figure, the accelerator comprises an attention calculation subsystem, a forward network calculation subsystem, an AXI read-write command control module, a first selector, a weight cache module. Figure 3

[0116] The attention calculation subsystem comprises a first input buffer module, a first addition module, a first normalization layer, a second selector, a multi-head attention module, and a sorting module.

[0117] The forward network calculation subsystem comprises a second input buffer module, a second addition module, a second normalization layer, and a forward network module.

[0118] The AXI read-write command control module sends the data received by the crossbar matrix to the attention calculation subsystem, the forward network calculation subsystem, and the weight cache module respectively.

[0119] The weight cache module receives the data from the AXI read-write command control module, caches the weight data required by the attention calculation subsystem and the forward network calculation subsystem, and sends the corresponding weight data to the attention calculation subsystem and the forward network calculation subsystem respectively.

[0120] The first input buffer module is used to cache the data read from the AXI read-write command control module, and sends the cached data to the first addition module; the first addition module performs addition operation on the cached data and sends the result to the first normalization layer, the second selector, and the first selector; the first normalization layer normalizes the data output by the first addition module and sends the result to the second selector; the second selector selects the data input to the multi-head attention module at the current time and sends the selected data to the multi-head attention module.

[0121] The multi-head attention module receives the selected data sent by the second selector and the weight data sent by the weight cache module, and performs multi-head attention mechanism calculation; the sorting module sorts and indexes the important tokens according to the importance scores of the tokens, prunes the unimportant tokens, and outputs the sorted data to the first selector.

[0122] The first selector selects the data input from the AXI read-write command control module at the current time and sends the selected data to the AXI read-write command control module.

[0123] ​The second input buffer module is used for buffering data read from the AXI read-write command control module, and sends the buffered data to the second addition module; the second addition module sends the buffered data to the second normalization layer after performing addition operation on the buffered data; the second normalization layer normalizes the data output by the second addition module and sends the normalized data to the forward network module and the AXI read-write command control module; the forward network module performs forward inference operation to obtain the calculation result of the current layer.

[0124] More specifically, let the time required for the forward network calculation subsystem and the attention calculation subsystem to complete one operation be △TF and △TA, respectively, then the average time required for the accelerator to complete one layer of operation is △T.

[0125] ΔT=max(ΔTF,△TA);

[0126] Let Q_space be the overall exploration space of the hardware design parameters, and Q_space_h be a set of design parameters that maximize the computing efficiency of the accelerator.

[0127] Q_space_h=argminQ|ΔTF-ΔTA|;

[0128] The calculation of the multi-head attention module is composed of attention mechanism calculation, splicing operation and matrix multiplication. When operating, the input vector matrix X is subjected to three matrix multiplication units to obtain the feature vector matrix Q, K and V, respectively; the feature vector matrix Q is multiplied by the transposed feature vector matrix K to obtain the importance score between each token through a normalized exponential function, and then multiplied by the feature vector matrix V to obtain the final output of the single-head attention mechanism.

[0129] △TQ, △TK, △TV, △TQK, △TMM are the calculation times of one run of the matrix multiplication of the attention mechanism.

[0130] Then we have:

[0131] ΔTA=max(ΔTQ,ΔTK,ΔTV,ΔTQK,ΔTMM)

[0132] Qsh=argminQ max(ΔTQ,ΔTK,ΔTV,ΔTQK,ΔTMM)

[0133] Qsh is the parameter set that minimizes the processing time ΔTA.

[0134] TQK is the multiplication of the feature vector matrix Q and the transposed feature vector matrix K.

[0135] TMM is the multiplication of the result of Q*K^T and V.

[0136] In this embodiment, as Figure 4As shown, the multi-head attention module further includes a feature vector calculation module.

[0137] During the calculation of the feature vector, the matrix slicing method is used to complete a matrix operation in a loop.

[0138] The matrix slicing method is as follows:

[0139] The input token matrix and the parameter matrix are cut into small matrices with dimensions [xh, xw] and [xw, ww] respectively; each small matrix is used as a basic calculation unit; each PE unit is composed of xw multipliers and a level addition tree, and a register is inserted between each multiplier and adder.

[0140] The entire PE array is composed of xh x ww PE units, and the calculation array can complete a small matrix multiplication in an average clock cycle when processing a continuous data stream.

[0141] The PE unit is a processing unit, and the PE array is a processing array.

[0142] In the design of the Q x KT attention score module, the input query and key block matrices will first be rearranged into the slicing size [xhqk, xwqk] and [xwqk, wwqk] of the module.

[0143] The Q x KT attention score module is responsible for Q x KT matrix multiplication calculation.

[0144] Pong cache is used to cache the data of the previous layer, reducing the calculation unit stall caused by matrix switching; the normalized exponential function input matrix is input into xhi x wwi natural exponential evaluation units by row, and the calculation results are input into the summation array to calculate the sum of the natural exponential of each row of data.

[0145] Pong cache refers to two storage spaces of the same size, one storing the current calculation data of the module, and the other storing the data for the next calculation of the module, so that the calculation flow does not stall due to waiting for calculation data.

[0146] The natural exponential result is written into the cache after rearrangement, and when the sum of each row of natural exponential is calculated, it is input into the division array to further complete the operation of the final result.

[0147] The summation array and the division array belong to the softmax function in the attention mechanism calculation. The attention mechanism calculation is composed of multiple attention heads.

[0148] The Matrix_Mul module is responsible for multiplication of two matrices; for implementation of the multi-head attention mechanism, a part of the attention head is implemented and the final calculation result is obtained by accumulating the intermediate results.

[0149] Since the bitonic sorting is a recursive algorithm, it is composed of multiple structural identical calculation layers, and each calculation layer is composed of multiple comparators. The full parallel bitonic sorting module refers to that each layer of the bitonic sorting is implemented on hardware, and it is a full pipelined full parallel calculation hardware structure, which can output an effective sorting result in each cycle. The folded bitonic sorting module refers to that multiple comparison operations of a calculation layer are folded to a comparator for calculation, and this implementation needs to insert a data rearrangement network between the calculation layers and reasonably control the direction of the comparator. Compared with the full parallel bitonic sorting module, the folded bitonic sorting module can effectively reduce the on-chip LUT resource occupation, but the cost is the increase of running time. In order to ensure that the calculation path does not need to wait for the sorting path, the running time of the sorting path needs to be less than that of the calculation path. The design of the folded bitonic sorting module meets this condition.

[0150] In the embodiment, the sorting module is a folded bitonic sorting module.

[0151] More specifically, the sorting module includes a sorting submodule and an index submodule.

[0152] The sorting submodule includes a first counter, a bitonic sequence generation module, and a bitonic sequence sorting unit.

[0153] The index submodule includes a second counter, a third counter, and a data-address index unit.

[0154] The first counter counts the number of input tokens and outputs the token sequence number corresponding to each token to the bitonic sequence generation module.

[0155] The bitonic sequence generation module generates a standard bitonic sequence and sends it to the bitonic sequence sorting unit for sorting.

[0156] Only the token importance score is sorted in the sorting process, and if an exchange occurs in the sorting process, the token importance score and the token sequence number are exchanged together.

[0157] The operation of the index submodule includes writing and reading.

[0158] The writing process is as follows:

[0159] The second counter selects the tokens output by the sorting part according to the pre-configured pruning retention number K, and writes the sequence numbers of the top K tokens with the largest importance into the data-address index unit; when the token calculation result is input, the third counter counts according to the token end signal token_in_end and outputs the token label; at this time, the token label is used to query the data-address index unit to determine whether the token needs to be retained; if the token needs to be retained, the calculation result of the token will be written into the off-chip DDR, otherwise it will not participate in subsequent operations.

[0160] The reading process is as follows:

[0161] According to the token label output by the third counter, the CAM is queried to read whether the corresponding address stores 0 or 1, if it is 1, the calculation result of the token is retained, and the calculation result is stored in the DDR; if it is 0, it is not stored and does not participate in subsequent operations.

[0162] More specifically, the hardware design space is explored based on the genetic algorithm, and the corresponding target function is:

[0163] Qh_param=argminQ max(ΔTF,ΔTA)=argminQΔT

[0164] Where Qh_param is the optimal design parameter, and Q is the search space of the overall design parameter.

[0165] According to the on-chip resource condition of the FPGA, there are different overall resource constraints, that is, the on-chip DSP, storage and bandwidth of the hardware design cannot exceed the highest value of the device; the utilization rate of the computing resource is used as the performance measurement index of the scheme.

[0166] The utilization rate of the computing resource is embodied as the ratio of the effective running time of the DSP unit to the overall running time.

[0167] Considering the influence of the running time of different layers, the DSE target function of the N-layer ViT network is modified as:

[0168] Qh=argmin QN∑i=1(ΔTi)

[0169] Through genetic algorithm search, the matrix cutting parameter of each matrix calculation module of the accelerator is obtained, and the optimal hardware accelerator design scheme is obtained.

[0170] Embodiment 3

[0171] In this embodiment, as Figure 7As shown, in the accelerator running, to complete the overall calculation of the visual self-attention model, taking two data sets to be processed as an example, after the accelerator starts, the MHA subsystem operates the attention mechanism part of the first layer visual self-attention model in the first data set in the first layer mode, and then caches the calculation result to the off-chip DDR, in the next period, the FFN subsystem starts and begins to process the output of the MHA subsystem in the last period. At the same time, the MHA subsystem starts to process the attention operation of the first layer of the second data set, and after the operation is completed, the FFN subsystem also completes the operation of the FFN part of the first layer visual self-attention model in the first data set, at this time, the MHA system can start to process the attention mechanism part of the second layer visual self-attention model in the first data set. After the accelerator completes the start, the FFN and MHA subsystems are running at the same time in each period, so as to fully utilize the computing performance of the system.

[0172] Obviously, the above embodiments of the present application are only examples for clearly illustrating the present application, and are not intended to limit the embodiments of the present application. Any modification, equivalent replacement and improvement within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. An FPGA-based visual self-attention model accelerator optimization method, characterized in that, The method comprises the following steps: The visual self-attention model is pruned by a dynamic token pruning scheme to remove redundant information, and an optimized visual self-attention model is obtained; When the visual self-attention model accelerator is constructed, the calculation process is divided by using a single visual self-attention calculation layer design on the FPGA and a matrix cutting method, and an optimal calculation resource allocation strategy is solved based on a genetic algorithm to maximize load balancing, so that an optimized visual self-attention model accelerator is obtained; the accelerator comprises an attention calculation subsystem, a forward network calculation subsystem, an AXI read-write command control module, a first selector and a weight cache module; The attention calculation subsystem comprises a first input buffer module, a first addition module, a first normalization layer, a second selector, a multi-head attention module and a sorting module; The forward network calculation subsystem comprises a second input buffer module, a second addition module, a second normalization layer and a forward network module; After the AXI read-write command control module receives data of the crossbar matrix, the data is sent to the attention calculation subsystem, the forward network calculation subsystem and the weight cache module respectively; After the weight cache module receives data of the AXI read-write command control module, the weight data required by the attention calculation subsystem and the forward network calculation subsystem are cached, and the corresponding weight data is sent to the attention calculation subsystem and the forward network calculation subsystem respectively; The first input buffer module is used to cache data read from the AXI read-write command control module, and the cached data is sent to the first addition module; the first addition module performs addition operation on the cached data and sends the data to the first normalization layer, the second selector and the first selector; the first normalization layer normalizes the data output by the first addition module and sends the data to the second selector; the second selector selects the data input to the multi-head attention module at the current time and sends the selected data to the multi-head attention module; The multi-head attention module receives the selected data sent by the second selector and the weight data sent by the weight cache module, and performs multi-head attention mechanism calculation; the sorting module sorts and indexes the tokens according to the importance scores of the tokens to complete the pruning of unimportant tokens, and outputs the sorted data to the first selector; The optimized visual self-attention model is run by using the optimized visual self-attention model accelerator.

2. The FPGA-based visual self-attention model accelerator optimization method of claim 1, wherein, The dynamic token pruning scheme is as follows: According to an output policy network in a reinforcement learning framework, a corresponding token pruning ratio is predicted for each layer of the visual self-attention model; The visual self-attention model sorts the tokens according to the pruning ratio and discards the tokens with the smallest importance; The pruning behavior of the current layer is expressed as a compression ratio, and the loss of the corresponding visual self-attention model under the pruning behavior in terms of accuracy is taken as the reward input of the scoring network, so as to update the integral network; Token pruning of the next layer of the visual self-attention model is performed until all layers are pruned.

3. The FPGA-based visual self-attention model accelerator optimization method of claim 2, wherein, For the dynamic token pruning scheme, an asynchronous advantage action evaluation algorithm is used to model the reward function and the optimal strategy of the agent respectively. The optimal strategy is the best pruning ratio for each layer; The framework of reinforcement learning consists of an environment and an agent; the environment is responsible for delivering the observation values and reward values in each state to the agent; the agent receives the observation values and reward values in each state from the environment for its own training, and performs task allocation of the visual self-attention model according to the received state variables; In the reinforcement learning environment, the state vector is returned after each pruning step, and the agent judges the changes in the environment and makes the next decision; For an N-layer visual self-attention model, an N+1-dimensional vector is used to represent the state of the visual self-attention model; When the agent runs to the ith step, that is, prunes to the ith layer, the state vector is defined as follows: where i / n is intended to indicate the current pruning layer for the agent, scaled to the range [0, 1]; represents the layer remaining token proportion.

4. The FPGA-based visual self-attention model accelerator optimization method of claim 2, wherein, The reward function scores the pruning effect of the agent and is used as part of the environment feedback to train the agent parameters, optimizing the behavior of the agent; The mathematical form of the reward function is as follows: The reward function contains two parts of compression ratio and accuracy loss, and contains two adjustable parameters λ and α; where λ factor is the scaling factor of the overall reward function, which scales the reward function value to [0, 1]; α is an importance adjustment factor, which adjusts the importance between the compression ratio and the accuracy loss, and then obtains different compression ratios; In the implementation process, the larger the α factor indicates that the agent tends to maintain the reduced accuracy loss, and vice versa.

5. The FPGA-based visual self-attention model accelerator optimization method of claim 1, wherein, The first selector selects the data input to the AXI read-write command control module at the current time, and sends the selected data to the AXI read-write command control module; The second input buffer module is used to buffer the data read from the AXI read-write command control module, and send the buffered data to the second addition module; the second addition module performs addition operation on the buffered data and sends it to the second normalization layer; the second normalization layer normalizes the data output by the second addition module and sends it to the forward network module and the AXI read-write command control module; the forward network module performs forward inference operation to obtain the calculation result of the current layer.

6. The FPGA-based visual self-attention model accelerator optimization method of claim 5, wherein, Let the time required for the forward network computation subsystem, the attention computation subsystem to complete one operation be TF, TA, then the average time required for the accelerator to complete one layer operation is T; T = max( TF, TA); Let Q_space be the overall exploration space of hardware design parameters, and Q_space_h be a set of design parameters that maximize the computing efficiency of the accelerator; ； The calculation of the multi-head attention module consists of attention mechanism calculation, splicing operation and matrix multiplication. When operating, the input vector matrix X is processed by three matrix multiplication units to obtain the feature vector matrix Q, K and V; The feature vector matrix Q and the feature vector matrix K are multiplied after transposition, and the importance score between tokens is obtained by the normalization exponential function, and then multiplied with the feature vector matrix V to obtain the final output of the single-head attention mechanism; TQ, TK, TV, TQK, TMM is the calculation time of each matrix multiplication operation of the attention mechanism, respectively; Then, TA = max( TQ, TK, TV, TQK, TMM) Qsh = argmin Qmax (Qsh, Qs, Qm, Qm) (1) TQ, TK, TV, TQK, TMM) Qsh is the minimum processing time TA parameter collection; TQK is the multiplication of the feature vector matrix Q and the feature vector matrix K after transposition; TMM is the multiplication of the result of QxKT and V.

7. The FPGA-based visual self-attention model accelerator optimization method of claim 5, wherein, The multi-head attention module further includes a feature vector calculation module; During feature vector calculation, the matrix slicing method is used to complete a matrix operation in a loop; The matrix slicing method is as follows: The input token matrix and the parameter matrix are cut into small matrices with dimensions of [xh, xw] and [xw, ww] respectively; each small matrix is taken as a basic calculation unit; each PE unit is composed of xw multipliers and a level addition tree, and a register is inserted between each multiplier and adder; The whole PE array is composed of xh×ww PE units, and the average one clock cycle is needed to complete a small matrix multiplication when the array processes continuous data stream; In the design of Q×KT attention score module, the input query and key block matrix will be first rearranged into the size of [xhqk, xwqk] and [xwqk, wwqk] of the module; Pong cache is used to cache the data of the previous layer, reducing the calculation unit stagnation caused by matrix switching; The normalized exponential function input matrix is input into xhi×wwi natural exponential evaluation units row by row, and the calculation results are input into the summation array to calculate the sum of the natural exponential of each row of data; The natural exponential result is written into the cache after rearrangement, and then input into the division array when the sum of the natural exponential of each row is calculated, and the final result is calculated. The Matrix_Mul module is responsible for the multiplication of two matrices, and the RAMMatrix_Mul module is responsible for the multiplication of two matrices.

8. The FPGA-based visual self-attention model accelerator optimization method of claim 5, wherein, The sorting module includes a sorting submodule, an index submodule; The sorting submodule includes a first counter, a double-tuned sequence generation module, and a double-tuned sequence sorting unit. The index submodule includes a second counter, a third counter, and a data-address index unit.

9. The FPGA-based visual self-attention model accelerator optimization method of claim 8, wherein, The first counter counts the number of input tokens and outputs the token sequence number corresponding to each token to the double-tuned sequence generation module. The double-tuned sequence generation module generates a standard double-tuned sequence and sends it to the double-tuned sequence sorting unit for sorting. Only the token importance score is sorted during the sorting process, and if an exchange occurs during the sorting process, the token importance score and the token sequence number are exchanged together. The operation of the index submodule includes writing and reading. The writing process is as follows: The second counter selects the tokens output by the sorting part according to the pre-configured pruning retention number K, and writes the sequence numbers corresponding to the top K tokens with the highest importance into the data-address index unit. When the token calculation result is input, the third counter counts and outputs the token label according to the token end signal token_in_end. If the token needs to be retained, the calculation result of the token will be written into the off-chip DDR, otherwise it will not participate in subsequent operations. The reading process is as follows:

10. The FPGA-based visual self-attention model accelerator optimization method of claim 5, wherein, According to the token label output by the third counter, query the CAM to read whether the address stored is 0 or 1. wherein are the optimal design parameters, Q is the search space of the overall design parameters; If it is 1, the calculation result of the token is retained and stored in the DDR; if it is 0, it is not stored and does not participate in subsequent operations. Based on genetic algorithm for hardware design space exploration, the corresponding target function is: According to the FPGA on-chip resource situation, there are different overall resource constraints, that is, the on-chip DSP, storage and bandwidth of the hardware design cannot exceed the maximum value of the device; the calculation resource utilization rate is used as the performance measurement index of the scheme. The computing resource utilization rate is embodied as a ratio of an effective running time of the DSP unit to an overall running time; Considering the influence of different layer running times, the ViT network DSE objective function of the Nth layer is modified; Through genetic algorithm search, the matrix cutting parameters of each matrix calculation module of the accelerator are obtained, and then the optimal hardware accelerator design scheme is obtained.

Images